search

canonical / snapcraft-rocks

Sources for Snapcraft as OCI images, maintained by the Starcraft team
GNU General Public License v3.0
4 stars 1 forks source link

Building snaps with `kde-neon` extension is not working #37

Open hhromic opened 2 months ago

hhromic commented 2 months ago

Hello, I recently found out about this repository and its intention to be the official way to build Snap packages in Docker. I tried to build our Snap packages using ghcr.io/canonical/snapcraft:8_core22 but sadly it did not work as expected.

The problem seems to be when using the kde-neon extension. Consider the following snap/snapcraft.yaml example adapted from the official guide for Qt5/KF applications:

name: kcalc
version: 19.08.0
grade: stable
adopt-info: kcalc
confinement: strict
base: core22

apps:
  kcalc:
    common-id: org.kde.kcalc.desktop
    command: kcalc
    extensions:
      - kde-neon
    plugs:
      - home
      - opengl
      - network
      - network-bind
      - pulseaudio

slots:
  session-dbus-interface:
    interface: dbus
    name: org.kde.kcalc.desktop
    bus: session

parts:
  kcalc:
    parse-info:
      - usr/share/metainfo/org.kde.kcalc.appdata.xml
    plugin: cmake
    build-snaps:
      - kf5-5-111-qt-5-15-11-core22-sdk
    build-packages:
      - libmpfr-dev
      - libgmp-dev
      - libkf5doctools-dev
    stage-packages:
      - libmpfr6
      - libgmp10
    source: https://download.kde.org/stable/applications/19.08.0/src/kcalc-19.08.0.tar.xz
    cmake-parameters:
      - "-DKDE_INSTALL_USE_QT_SYS_PATHS=ON"
      - "-DCMAKE_INSTALL_PREFIX=/usr"
      - "-DCMAKE_BUILD_TYPE=Release"
      - "-DENABLE_TESTING=OFF"
      - "-DBUILD_TESTING=OFF"
      - "-DKDE_SKIP_TEST_SETTINGS=ON"

The above Snapcraft file does not build using the ghcr.io/canonical/snapcraft:8_core22 image:

$ docker run -it -v `pwd`:/project ghcr.io/canonical/snapcraft:8_core22
2024-04-28T21:36:27.568Z [pebble] Started daemon.
2024-04-28T21:36:27.598Z [pebble] POST /v1/services 18.095676ms 202
2024-04-28T21:36:27.599Z [pebble] Started default services with change 1.
2024-04-28T21:36:27.610Z [pebble] Service "snapcraft" starting: /bin/run-snapcraft.sh [ pack ]
2024-04-28T21:36:27.634Z [snapcraft]
2024-04-28T21:36:27.635Z [snapcraft] WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
2024-04-28T21:36:27.635Z [snapcraft]
2024-04-28T21:36:27.925Z [snapcraft] Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
2024-04-28T21:36:27.983Z [snapcraft] Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
2024-04-28T21:36:28.491Z [snapcraft] Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
2024-04-28T21:36:28.640Z [snapcraft] Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
2024-04-28T21:36:28.787Z [snapcraft] Get:5 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
2024-04-28T21:36:28.939Z [snapcraft] Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
2024-04-28T21:36:29.099Z [snapcraft] Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1755 kB]
2024-04-28T21:36:32.254Z [snapcraft] Get:8 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.7 kB]
2024-04-28T21:36:32.291Z [snapcraft] Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2265 kB]
2024-04-28T21:36:32.380Z [snapcraft] Get:10 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
2024-04-28T21:36:32.391Z [snapcraft] Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
2024-04-28T21:36:32.705Z [snapcraft] Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2035 kB]
2024-04-28T21:36:33.301Z [snapcraft] Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1370 kB]
2024-04-28T21:36:33.528Z [snapcraft] Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2339 kB]
2024-04-28T21:36:33.832Z [snapcraft] Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [51.1 kB]
2024-04-28T21:36:33.846Z [snapcraft] Get:16 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [81.0 kB]
2024-04-28T21:36:33.846Z [snapcraft] Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [31.9 kB]
2024-04-28T21:36:34.320Z [snapcraft] Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1077 kB]
2024-04-28T21:36:37.034Z [snapcraft] Fetched 31.4 MB in 9s (3359 kB/s)
2024-04-28T21:36:37.034Z [snapcraft] Reading package lists...
2024-04-28T21:36:39.910Z [snapcraft] Building dependency tree...
2024-04-28T21:36:40.215Z [snapcraft] Reading state information...
2024-04-28T21:36:40.273Z [snapcraft] 15 packages can be upgraded. Run 'apt list --upgradable' to see them.
2024-04-28T21:36:45.918Z [snapcraft] Starting Snapcraft 8.0.5
2024-04-28T21:36:45.918Z [snapcraft] Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20240428-213645.910384.log'
2024-04-28T21:36:45.957Z [snapcraft] Running on amd64 for amd64
2024-04-28T21:36:45.966Z [snapcraft] Initializing parts lifecycle
2024-04-28T21:36:45.981Z [snapcraft] Failed to pull source: unable to determine source type of '/usr/share/snapcraft/extensions/desktop/kde-neon'.
2024-04-28T21:36:45.987Z [snapcraft] Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240428-213645.910384.log'
2024-04-28T21:36:46.344Z [pebble] Service "snapcraft" stopped unexpectedly with code 1
2024-04-28T21:36:46.344Z [pebble] Service "snapcraft" on-failure action is "shutdown", triggering failure shutdown
2024-04-28T21:36:46.344Z [pebble] Server exiting!

The same Snapcraft file does start building correctly in Ubuntu with the regular snapcraft command:

$ snapcraft --use-lxd --verbose
Starting Snapcraft 8.0.5
Logging execution to '/home/hhromic/.local/state/snapcraft/log/snapcraft-20240428-223554.190257.log'
Running on amd64 for amd64
Launching instance...
Starting instance
Starting Snapcraft 8.0.5
Logging execution to '/tmp/snapcraft.log'
Running on amd64 for amd64
Initializing parts lifecycle
Installing build-packages
Installing build-snaps
Pulling kcalc
Fetching stage-packages
Pulling kde-neon/sdk
Building kcalc
:: + cmake /root/parts/kcalc/src -G 'Unix Makefiles' -DKDE_INSTALL_USE_QT_SYS_PATHS=ON -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DENABLE_TESTING=OFF -DBUILD_TESTING=OFF -DKDE_SKIP_TEST_SETTINGS=ON
:: -- The C compiler identification is GNU 11.2.0
:: -- The CXX compiler identification is GNU 11.2.0
:: -- Detecting C compiler ABI info
:: -- Detecting C compiler ABI info - done
:: -- Check for working C compiler: /usr/bin/cc - skipped
:: -- Detecting C compile features
:: -- Detecting C compile features - done
:: -- Detecting CXX compiler ABI info
:: -- Detecting CXX compiler ABI info - done
...

Warning: The kcalc build itself does not actually succeed in this example, but at least it does start building.

Is there something I'm doing wrong when using the ghcr.io/canonical/snapcraft:8_core22 image or Snapcraft files using extensions is not yet supported by the images in this repository?

tigarmo commented 2 months ago

Hi, thanks for the report!

Is there something I'm doing wrong when using the ghcr.io/canonical/snapcraft:8_core22 image or Snapcraft files using extensions is not yet supported by the images in this repository?

The later, I think. Your usage seems correct but we probably need to do some work to support extensions (in particular those that need sdks). The usage of build-snaps is possibly also problematic but I'll look into it.

hhromic commented 2 months ago

Thanks for replying and looking into it!

As further information, I also tried building a Docker image based on the older Dockerfile for running Snapcraft in Docker.

The below is the Dockerfile I made based on the above instructions:

ARG RISK=stable
ARG UBUNTU=jammy

FROM ubuntu:$UBUNTU as builder
ARG RISK
ARG UBUNTU
RUN echo "Building snapcraft:$RISK in ubuntu:$UBUNTU"

# Grab dependencies
RUN apt-get update
RUN apt-get dist-upgrade --yes
RUN apt-get install --yes \
      curl \
      jq \
      squashfs-tools

# Grab the core22 snap (which snapcraft uses as a base) from the stable channel
# and unpack it in the proper place.
RUN curl -L $(curl -H 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/core22' | jq '.download_url' -r) --output core22.snap
RUN mkdir -p /snap/core22
RUN unsquashfs -d /snap/core22/current core22.snap

# Grab the snapcraft snap from the $RISK channel and unpack it in the proper
# place.
RUN curl -L $(curl -H 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel='$RISK | jq '.download_url' -r) --output snapcraft.snap
RUN mkdir -p /snap/snapcraft
RUN unsquashfs -d /snap/snapcraft/current snapcraft.snap

# Fix Python3 installation: Make sure we use the interpreter from
# the snapcraft snap:
RUN unlink /snap/snapcraft/current/usr/bin/python3
RUN ln -s /snap/snapcraft/current/usr/bin/python3.* /snap/snapcraft/current/usr/bin/python3
RUN echo /snap/snapcraft/current/lib/python3.*/site-packages >> /snap/snapcraft/current/usr/lib/python3/dist-packages/site-packages.pth

# Create a snapcraft runner (TODO: move version detection to the core of
# snapcraft).
RUN mkdir -p /snap/bin
RUN echo "#!/bin/sh" > /snap/bin/snapcraft
RUN snap_version="$(awk '/^version:/{print $2}' /snap/snapcraft/current/meta/snap.yaml | tr -d \')" && echo "export SNAP_VERSION=\"$snap_version\"" >> /snap/bin/snapcraft
RUN echo 'exec "$SNAP/usr/bin/python3" "$SNAP/bin/snapcraft" "$@"' >> /snap/bin/snapcraft
RUN chmod +x /snap/bin/snapcraft

# Multi-stage build, only need the snaps from the builder. Copy them one at a
# time so they can be cached.
FROM ubuntu:$UBUNTU
COPY --from=builder /snap/core22 /snap/core22
COPY --from=builder /snap/snapcraft /snap/snapcraft
COPY --from=builder /snap/bin/snapcraft /snap/bin/snapcraft

# Generate locale and install dependencies.
RUN apt-get update && apt-get dist-upgrade --yes && apt-get install --yes snapd sudo locales && locale-gen en_US.UTF-8

# Set the proper environment.
ENV LANG="en_US.UTF-8"
ENV LANGUAGE="en_US:en"
ENV LC_ALL="en_US.UTF-8"
ENV PATH="/snap/bin:/snap/snapcraft/current/usr/bin:$PATH"
ENV SNAP="/snap/snapcraft/current"
ENV SNAP_NAME="snapcraft"
ENV SNAP_ARCH="amd64"
ENV SNAPCRAFT_BUILD_ENVIRONMENT="host"

When I try to build the Snap package with the above image, I obtain the same error as with Snapcraft Rocks:

$ docker run --rm -it -v $PWD:/build -w /build snapcraft
root@0c5adfd2f8ab:/build# snapcraft
Failed to pull source: unable to determine source type of '/snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon'.
Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240430-204629.516750.log'

The execution log is below:

2024-04-30 20:46:29.518 Starting snapcraft, version 8.2.1
2024-04-30 20:46:29.519 Log verbosity level set to BRIEF
2024-04-30 20:46:29.527 Falling back from craft-application to snapcraft.
2024-04-30 20:46:29.530 lifecycle command: 'pack', arguments: Namespace(destructive_mode=False, use_lxd=False, debug=False, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)
2024-04-30 20:46:29.530 command: pack, arguments: Namespace(destructive_mode=False, use_lxd=False, debug=False, enable_manifest=False, manifest_image_information=None, bind_ssh=False, build_for=None, http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None)
2024-04-30 20:46:29.546 CPU count (from process affinity): 4
2024-04-30 20:46:29.546 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT ''
2024-04-30 20:46:29.546 Running on amd64 for amd64
2024-04-30 20:46:29.551 Initializing parts lifecycle
2024-04-30 20:46:29.551 is_snap: True, SNAP_NAME set to snapcraft
2024-04-30 20:46:29.555 process kcalc:Step.PULL
2024-04-30 20:46:29.555 add action kcalc:Step.PULL(ActionType.RUN)
2024-04-30 20:46:29.556 process kde-neon/sdk:Step.PULL
2024-04-30 20:46:29.556 add action kde-neon/sdk:Step.PULL(ActionType.RUN)
2024-04-30 20:46:29.557 process kcalc:Step.BUILD
2024-04-30 20:46:29.557 add action kcalc:Step.BUILD(ActionType.RUN)
2024-04-30 20:46:29.558 process kde-neon/sdk:Step.BUILD
2024-04-30 20:46:29.558 add action kde-neon/sdk:Step.BUILD(ActionType.RUN)
2024-04-30 20:46:29.558 process kcalc:Step.STAGE
2024-04-30 20:46:29.558 add action kcalc:Step.STAGE(ActionType.RUN)
2024-04-30 20:46:29.559 process kde-neon/sdk:Step.STAGE
2024-04-30 20:46:29.559 add action kde-neon/sdk:Step.STAGE(ActionType.RUN)
2024-04-30 20:46:29.560 process kcalc:Step.PRIME
2024-04-30 20:46:29.560 add action kcalc:Step.PRIME(ActionType.RUN)
2024-04-30 20:46:29.561 process kde-neon/sdk:Step.PRIME
2024-04-30 20:46:29.561 add action kde-neon/sdk:Step.PRIME(ActionType.RUN)
2024-04-30 20:46:29.564 part build packages: ['libmpfr-dev', 'libgmp-dev', 'libkf5doctools-dev']
2024-04-30 20:46:29.564 source build packages: {'tar'}
2024-04-30 20:46:29.564 plugin build packages: {'cmake', 'gcc'}
2024-04-30 20:46:29.564 part build snaps: ['kf5-5-111-qt-5-15-11-core22-sdk']
2024-04-30 20:46:29.565 Failed to pull source: unable to determine source type of '/snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon'.
2024-04-30 20:46:29.567 Traceback (most recent call last):
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/snapcraft/application.py", line 361, in main
2024-04-30 20:46:29.567     return app.run()
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_application/application.py", line 481, in run
2024-04-30 20:46:29.567     dispatcher = self._get_dispatcher()
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/snapcraft/application.py", line 220, in _get_dispatcher
2024-04-30 20:46:29.567     raise errors.ClassicFallback()
2024-04-30 20:46:29.567 snapcraft.errors.ClassicFallback
2024-04-30 20:46:29.567
2024-04-30 20:46:29.567 During handling of the above exception, another exception occurred:
2024-04-30 20:46:29.567 Traceback (most recent call last):
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/snapcraft/parts/parts.py", line 207, in run
2024-04-30 20:46:29.567     with self._lcm.action_executor() as aex:
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 307, in __enter__
2024-04-30 20:46:29.567     self._executor.prologue()
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 89, in prologue
2024-04-30 20:46:29.567     self._install_build_packages()
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 239, in _install_build_packages
2024-04-30 20:46:29.567     self._create_part_handler(part)
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 224, in _create_part_handler
2024-04-30 20:46:29.567     handler = PartHandler(
2024-04-30 20:46:29.567   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 111, in __init__
2024-04-30 20:46:29.568     self._source_handler = sources.get_source_handler(
2024-04-30 20:46:29.568   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/sources/sources.py", line 131, in get_source_handler
2024-04-30 20:46:29.568     handler_class = _get_source_handler_class(
2024-04-30 20:46:29.568   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/sources/sources.py", line 162, in _get_source_handler_class
2024-04-30 20:46:29.568     source_type = get_source_type_from_uri(source)
2024-04-30 20:46:29.568   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/sources/sources.py", line 199, in get_source_type_from_uri
2024-04-30 20:46:29.568     raise errors.InvalidSourceType(source)
2024-04-30 20:46:29.568 craft_parts.sources.errors.InvalidSourceType: Failed to pull source: unable to determine source type of '/snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon'.
2024-04-30 20:46:29.568 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240430-204629.516750.log'

Hope this helps debug further and hope this eventually starts working in the newer Rocks images :)

hhromic commented 2 months ago

A-ha! I just noticed from the above log that the path Snapcraft is trying to use is:

/snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon

However, the extension is actually installed in here (notice no /usr/ beween current and share):

/snap/snapcraft/current/share/snapcraft/extensions/desktop/kde-neon

For the sake of testing I manually copied the contents from /snap/snapcraft/current/share/snapcraft/extensions/desktop/kde-neon into /snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon and the source type error is gone!

The example build then fails at the same place as in a regular Snapcraft installation in Ubuntu (as expected). I went to check in my testing Ubuntu VM and the extension is also in /snap/snapcraft/11471/share/snapcraft/extensions/desktop/kde-neon, so there must be a path configuration issue somewhere in Snapcraft itself.

Will try to investigate more where is the actual problem.

hhromic commented 2 months ago

Looking more into this now.

I now learned about the snapcraft expand-extensions command. On my testing Ubuntu VM it resolves correctly:

    kde-neon/sdk:
        source: /snap/snapcraft/11471/share/snapcraft/extensions/desktop/kde-neon
        plugin: make
        make-parameters:
        - PLATFORM_PLUG=kf5-5-111-qt-5-15-11-core22

In the (old) Docker-based Snapcraft it does not resolve correctly:

    kde-neon/sdk:
        source: /snap/snapcraft/current/usr/share/snapcraft/extensions/desktop/kde-neon
        plugin: make
        make-parameters:
        - PLATFORM_PLUG=kf5-5-111-qt-5-15-11-core22

And in the Snapcraft Rocks image expands to this:

$ docker run -it -v `pwd`:/project ghcr.io/canonical/snapcraft:8_core22 expand-extensions
(...)
2024-04-30T21:18:06.492Z [snapcraft]     kde-neon/sdk:
2024-04-30T21:18:06.492Z [snapcraft]         source: /usr/share/snapcraft/extensions/desktop/kde-neon
2024-04-30T21:18:06.492Z [snapcraft]         plugin: make
2024-04-30T21:18:06.492Z [snapcraft]         make-parameters:
2024-04-30T21:18:06.492Z [snapcraft]         - PLATFORM_PLUG=kf5-5-111-qt-5-15-11-core22
(...)

In the Snapcraft Rocks image, the extensions seem to be installed in /share/snapcraft (at the root), which seems incorrect:

$ docker run -it -v `pwd`:/project --entrypoint bash ghcr.io/canonical/snapcraft:8_core22 -c 'ls -la /share/snapcraft'
total 20
drwxr-xr-x 5 root root 4096 Apr  2 00:46 .
drwxr-xr-x 4 root root 4096 Apr  2 00:46 ..
drwxr-xr-x 5 root root 4096 Apr  2 00:46 extensions
drwxr-xr-x 2 root root 4096 Apr  2 00:46 keyrings
drwxr-xr-x 2 root root 4096 Apr  2 00:46 schema
hhromic commented 2 months ago

Alright, digging deeper, I think the issue is in how Snapcraft searches for extensions.

When expanding extensions, the following function is used by the extensions in Snapcraft: https://github.com/canonical/snapcraft/blob/b8ff64b89dfe19fc1cd215308b4dff8b70b7f4ad/snapcraft/extensions/extension.py#L129-L131

As it can be seen, this is hard-coded to the Python interpreter's system prefix:

return Path(sys.prefix) / "share" / "snapcraft" / "extensions"

In the case of the old-Docker Snapcraft image, the Python interpreter is set as:

RUN echo 'exec "$SNAP/usr/bin/python3" "$SNAP/bin/snapcraft" "$@"' >> /snap/bin/snapcraft

Which results in sys.prefix being /snap/snapcraft/current/usr (not where the extensions are installed).

However, the Snapcraft own Snap metadata uses bin/python (instead of usr/bin/python):

apps:
  snapcraft:
    environment:
      # https://github.com/lxc/pylxd/pull/361
      PYLXD_WARNINGS: "none"
    command: bin/python $SNAP/bin/snapcraft
    completer: snapcraft-completion

If I update the old-Dockerfile accordingly to match the above:

RUN echo 'exec "$SNAP/bin/python" "$SNAP/bin/snapcraft" "$@"' >> /snap/bin/snapcraft

Then it works correctly! 🎉

For the Snapcraft Rocks image, the Python interpreter used is installed in /usr, and therefore sys.prefix is /usr.

In summary, I'm convinced now that the Snapcraft Rocks image should have the Snapcraft data installed in /usr/share and not in /share (root of the container).

EDIT: Sorry for all the spam 🙈.