Build of Docker Ubuntu 20.10 image fails

[emulti]

Trying to compile using Docker as shown in your Wiki. Unfortunately building the Docker image for Ubuntu 20.10 fails due to lack of Release data on security.ubuntu.com/archive.ubuntu.com. The package files seem to have been deleted because 20.10 is no longer supported.

Reading package lists... E: The repository 'http://security.ubuntu.com/ubuntu groovy-security Release' does not have a Release file. E: The repository 'http://archive.ubuntu.com/ubuntu groovy Release' does not have a Release file. E: The repository 'http://archive.ubuntu.com/ubuntu groovy-updates Release' does not have a Release file. E: The repository 'http://archive.ubuntu.com/ubuntu groovy-backports Release' does not have a Release file. The command '/bin/sh -c apt-get update && apt-get install --assume-yes wget git' returned a non-zero code: 100

I tried using other Ubuntu stable versions without the patch in docker/linux-image.sh.diff but without success, also compiling using the standard Cargo docker image, but without success.

What is the current image you are using to build TubeFeeder?

[Schmiddiii]

I honestly completly forgot about the docker image. For development I just compile for x86 for my computer (just cargo run if Rust is installed), and if I want to test in on my PinePhone I publish a flathub-beta build. You can also just compile the flatpak (it is also able to cross-compile, but will take some time).

If you want I will look at the docker build tomorrow, otherwise I will officially deprecate the image from the repo and the wiki.

[emulti]

Thanks for responding! I was able to build on x86_64 with 'cargo run'. It's necessary to use a specific 22-03-22 nightly Rust toolchain at the moment due to a regression compiling the 'ring' crate.

Cross-compiling, I tried using Ubuntu 21.04 and 18.04 instead of 20.10 in the Docker image, but both failed to build, probably because I don't understand what is being done in the patch to linux-image.sh

Also a cross-compile with the standard "cross" docker image didn't work. This has worked for me with Delta Chat but it fails with TF. Maybe my inexperience, however.

The flathub stable build installs fine on the Pinephone. However it and the x86 build from Master both seem unable to add a YouTube subscription at present- no results are found either with channel name or ID. LBRY works, however!

On 26 March 2022 20:15:42 GMT, Schmiddiii @.***> wrote:

I honestly completly forgot about the docker image. For development I just compile for x86 for my computer (just cargo run if Rust is installed), and if I want to test in on my PinePhone I publish a flathub-beta build. You can also just compile the flatpak (it is also able to cross-compile, but will take some time).

If you want I will look at the docker build tomorrow, otherwise I will officially deprecate the image from the repo and the wiki.

-- Reply to this email directly or view it on GitHub: https://github.com/Tubefeeder/Tubefeeder/issues/46#issuecomment-1079767353 You are receiving this because you authored the thread.

Message ID: @.***>

[Schmiddiii]

It's necessary to use a specific 22-03-22 nightly Rust toolchain at the moment due to a regression compiling the 'ring' crate.

Weird, I am on Rust stable (rustc 1.57.0 (f1edd0429 2021-11-29)) and can build fine. Maybe update rust?

Cross-compiling, I tried using Ubuntu 21.04 and 18.04 instead of 20.10 in the Docker image, but both failed to build, probably because I don't understand what is being done in the patch to linux-image.sh

I must honestly say I forget what exactly this patch does, but I believe it added some packages and updated the kernel.

Also a cross-compile with the standard "cross" docker image didn't work. This has worked for me with Delta Chat but it fails with TF. Maybe my inexperience, however.

Yes, as this needs, as far as I remember, some libraries to link to during compilation (probably gtk and similar, ssl) where you probably to not have the aarch-version installed (I do not recommend installing these, that is what the docker was for).

However it and the x86 build from Master both seem unable to add a YouTube subscription at present- no results are found either with channel name or ID

I cannot reproduce this, but I have noticed some issues yesterday with the backend I am using for YouTube (the API of piped.kavin.rocks threw some bad gateway errors). Maybe this was the issue. Please try again, otherwise please consider sending the log using RUST_LOG=debug cargo run or RUST_LOG=debug flatpak run de.schmidhuberj.tubefeeder.

Do you need to cross-compile Tubefeeder or would you also accept just using the Flatpak, or cross-compiling the flatpak? Depending on your answer I would either work on the docker again (with no guarantee of success, or how long it would take) or just deprecate this form of compilation.

[emulti]

You were right about updating Rust Stable, I hadn't done it for a while and it now compiles fine on x86_64.

Ah, I see you are now using Piped rather than yt-dlp/youtube-dl as the Wiki indicates! However it's still not working for me. Adding a YT subscription gives the following log. Looks like something minor in the parsing of the Piped response.

2022-03-27T12:21:28Z DEBUG tubefeeder::csv_file_manager] A error opening the file occured [2022-03-27T12:21:28Z DEBUG tf_playlist::playlist_manager] Creating new playlist for observer [2022-03-27T12:21:28Z DEBUG tf_playlist::playlist_manager] Adding observer to existing playlist [2022-03-27T12:21:28Z DEBUG tubefeeder::csv_file_manager] A error opening the file occured [2022-03-27T12:21:28Z DEBUG tubefeeder::gui::feed::feed_page::imp] Reloading [2022-03-27T12:21:28Z DEBUG tubefeeder::csv_file_manager] A error opening the file occured [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Starting getting subscriptions [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Finished getting subscriptions [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Starting getting subscriptions [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Finished getting subscriptions [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Starting getting subscriptions [2022-03-27T12:21:28Z DEBUG tf_core::pipeline::merger] Finished getting subscriptions [2022-03-27T12:21:43Z DEBUG tf_platform_youtube::subscription] Getting channel from query CuriousMarc [2022-03-27T12:21:43Z DEBUG reqwest::connect] starting new connection: https://pipedapi.kavin.rocks/ [2022-03-27T12:21:45Z DEBUG reqwest::async_impl::client] response '200 OK' for https://pipedapi.kavin.rocks/search?q=CuriousMarc&filter=channels [2022-03-27T12:21:45Z ERROR tf_platform_youtube::subscription] Got back a error: missing field name at line 1 column 4733 [2022-03-27T12:21:45Z ERROR tubefeeder::gui::subscription::subscription_page::imp] Failed to get subscription with supplied data

Due to the limited Emmc capacity on Pinephone I try to (cross) compile where possible rather than using Flatpak and co. Flatpak installs are also harder to customise (for example on PP I find 360p/480p videos are fine to watch, save bandwidth and download time, also disk space). Updating the Docker build image would therefore be welcome if you have the time, though if using Ubuntu it might better to wait until 22.04 is released? In the meantime I will have a shot at making it work with an Archlinux image (my preferred platform) as a learning exercise.

[Schmiddiii]

I am using Piped for querying the videos from a subscription, and the subscription from the name. Depending on the player (mpv by default), yt-dlp/youtube-dl will be used to play the video (so one would not even need yt-dlp/youtube-dl for running the application).

The log looks pretty interesting. I will try to fix this. In the meantime, if you have the ID of the channel, you can just add the line youtube,<channel-id> to ~/.local/share/tubefeeder/subscriptions.csv.

Update: Seems to be a issue with Piped, as this sometimes returns videos when querying for channels. I have already contacted the owner of Piped on this regard.

If you succeed in cross-compiling the application using Arch, feel free to share what you have done and maybe push the used docker-file and instructions.

[emulti]

Thanks for the information above. I started to look at using an Arch docker image- it looks far beyond my capabilities, cross seems to be based on Ubuntu/Debian/Centos images.

[Schmiddiii]

I have opened a PR that should fix the compilation. The needed commands are listed in the PR, the result is in target/release/tubefeeder. Note that you will also have to install internationalization files (everything in po/locale with ending .mo to /usr/share/locale with the same directory structure). I will update the documentation after the PR is merged.

The problem with the subscriptions seems to have been a temporary problem with Piped, it displayed video results when searching for channels resulting in failed deserialization. It seemed to be location-dependent on whether this issue actually arises (probably something wrong with some YouTube server). Last I checked, the issue was not there anymore.

[emulti]

Thanks- I will test at the weekend after the merge

Subscription adding still seems to be hit and miss. The log shows 'n' results are obtained from api.kavin.rocks each time, but the sub (the first result?) isn't added. The log is the same for successful and unsuccessful adds. I also noticed that adding LBRY subs doesn't always work, though the channel works OK if manually added to subscriptions.csv

[Schmiddiii]

If you would not mind, I would like you to test the PR-branch before merging. I do not want to merge anything broken. Testing it in the weekend will be enough.

Regarding the subscriptions, I found a pretty severe bug that would only allow one subscription at a time. This might have been your problem (it did not log that it failed to add the subscription), but Youtube and LBRY were handled the same, so I am not sure if this exactly your problem. See PR #48.

[emulti]

The docker image rust-with-libadwaita builds fine with the docker commands in the PR. The cargo command to compile (with rust-with-libadwaita as the image name rather than rust-with-gtk) in the PR gives:

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested

It then proceeds to cross-compile the rust crates using qemu, and you are right that it's slow: after 7 hours (quad core i7, 8GB ram and 4GB swap, all 99% used) it had still not finished. At that point I gave up with that method.

Alternative try: put the image name in Cross.toml and run cross build --target aarch64-unknown-linux-gnu --release

The first time I ran this I omitted the --release and it built in 265 minutes, with much less use of RAM and swap. The executable for Tubefeeder was 288M in size... I will do another run without the debug symbols and try and figure out how to avoid rebuilding all the crates each run, if that is possible. Is there a way to avoid cross-compilation using qemu on the host, by having a cross toolchain in the image?

[Schmiddiii]

The docker image rust-with-libadwaita builds fine with the docker commands in the PR. The cargo command to compile (with rust-with-libadwaita as the image name rather than rust-with-gtk) in the PR gives:

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested

I also got that warning, but it is not detremental to the build.

It then proceeds to cross-compile the rust crates using qemu, and you are right that it's slow: after 7 hours (quad core i7, 8GB ram and 4GB swap, all 99% used) it had still not finished. At that point I gave up with that method.

I did not think it would take that long, it took maybe 3 hours for me in a system with approximatly equal stats.

Alternative try: put the image name in Cross.toml and run cross build --target aarch64-unknown-linux-gnu --release

The first time I ran this I omitted the --release and it built in 265 minutes, with much less use of RAM and swap. The executable for Tubefeeder was 288M in size...

That makes no sense that cross works with this image. I chose to not go with cross on this attempt as I was not able to compile the docker image with the new Ubuntu version. But if it works, that is great (even though I have no idea how it is able to work). Did you test the executable on your PinePhone?

I will do another run without the debug symbols and try and figure out how to avoid rebuilding all the crates each run, if that is possible. Is there a way to avoid cross-compilation using qemu on the host, by having a cross toolchain in the image?

I do not think sharing the target would be easily possible, but one may be able to extract the target folder from the docker container and mount it in the next run. But that will not help if the first compilation is debug and the second release. I do not think you can avoid having qemu easily, as, like previously mentioned, the compilation requires some libraries to link to, and you likely do not have them installed in the aarch64-platform. You might be able to install these, but I think it would not be easy and I have not yet done that.

Sorry that cross compiling this application is so hard. I have not done that for a while myself. What I have also thought of what might be possible is compile it on a RasberryPi if you have one around (I do not know if this would work). Another possibility would be to download the aarch-flatpak and extract the binary somehow (I have no idea how this works, but I think I saw somewhere that it might work, and I have no guarantee that the binary would also work outside the flatpak).

[emulti]

The executable built with cross ran fine on the PP after putting the locale files in place. It's notable that a Pinephone struggles to play resolutions over 480p without dropping frames. But over four hours to build with docker/qemu is a bit impractical. I tried a 'normal' cross-compile with an aarch64 toolchain, but it fails at the final linking step, because the system libraries for the target architecture are not available on the host.

[Schmiddiii]

It's notable that a Pinephone struggles to play resolutions over 480p without dropping frames

I would recomment Clapper as a video player as it has hardware acceleration on the PP.

But over four hours to build with docker/qemu is a bit impractical.

I totally aggree. I will maybe test around more with this and maybe try to install the needed dependencies in aarch-version and then compile on a x86-docker. But I have no idea whether I will be able to make it work.

[Schmiddiii]

I have tried around a little more to speed up cross compilation without qemu and must conclude that it seems impossible to do:

Compilation indirectly needs gobject-introspection:arm64, which indirectly depends on build-essential:arm64. This for some reason needs to install binutils:arm64, which is in conflict with binutils:amd64. And binutils:amd64 does not seem easily uninstallable.

This explanation is probably very confusing, but I do not know how I could describe it better. I have tried around for the last few hours and was not able to circumvent it. Cross-compiling sucks.

I hope you are ok with the long compilation times, but I really do not know how they could be significantly improved. I will merge the PR and write the documentation about cross-compilation tomorrow.

[emulti]

I attempted cross-compilation on Arch with a cross-toolchain installed. It takes about 15 minutes, though obviously fails at the linking step when the aarch64 libraries cannot be found on the x86_64 host. This makes me hopeful that a cross-compilation in Docker could be made to work, using an x86_64 image that supports the installation of libraries from different architectures (Debian, Ubuntu?). Or are your remarks above based on attempts with Debian? Maybe I am wrong in this? Cross-compilation is very confusing. For Rust applications, maybe it's time to set up an aarch64 build machine. I have some suitable SBC laying around.

[Schmiddiii]

My tests were on Debian, but trying to install the needed libraries to compile leads to package conflicts that I was not able to install. I therefore belive that cross-compiling GTK-apps with compiletime-dependencies is likely impossible, independent of the language (If you find a counter-example, please inform me, it might help to solve the problem).

My last attempt, failes at installing build-essential if I remember correctly

``` FROM buildpack-deps:bookworm # First two "commands" copied from https://github.com/rust-lang/docker-rust/blob/c53032d5763aa335e338d118d1bfd6c5ecc09e31/1.59.0/buster/Dockerfile ENV RUSTUP_HOME=/usr/local/rustup \ CARGO_HOME=/usr/local/cargo \ PATH=/usr/local/cargo/bin:$PATH \ RUST_VERSION=1.59.0 RUN set -eux; \ dpkgArch="$(dpkg --print-architecture)"; \ rustArch='x86_64-unknown-linux-gnu'; \ rustupSha256='3dc5ef50861ee18657f9db2eeb7392f9c2a6c95c90ab41e45ab4ca71476b4338'; \ url="https://static.rust-lang.org/rustup/archive/1.24.3/${rustArch}/rustup-init"; \ wget "$url"; \ echo "${rustupSha256} *rustup-init" | sha256sum -c -; \ chmod +x rustup-init; \ ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \ rm rustup-init; \ chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \ rustup --version; \ cargo --version; \ rustc --version; RUN apt update RUN apt upgrade --yes RUN apt install --yes gcc-aarch64-linux-gnu libxml2-utils RUN rustup target add aarch64-unknown-linux-gnu RUN dpkg --add-architecture arm64 RUN apt update RUN apt install --yes libadwaita-1-0:arm64 libssl-dev:arm64 libglib2.0-dev:arm64 RUN git clone https://github.com/Tubefeeder/Tubefeeder.git WORKDIR Tubefeeder RUN rm build.rs RUN glib-compile-resources resources.gresource.xml ENV PKG_CONFIG_LIBDIR=/usr/local/lib/aarch64-linux-gnu/pkgconfig:/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig/ \ PKG_CONFIG_ALLOW_CROSS=1 RUN apt install --yes libpango1.0-dev:arm64 libgdk-pixbuf-2.0-dev:arm64 RUN apt remove --yes binutils build-essential dpkg-dev RUN apt install --yes make:arm64 RUN apt install --yes binutils:arm64 RUN apt install --yes dpkg-dev:arm64 RUN apt install --yes make:arm64 dpkg-dev:arm64 build-essential:arm64 --no-install-recommends RUN apt install --yes make:arm64 binutils:arm64 build-essential:arm64 gobject-introspection:arm64 --no-install-recommends RUN apt-get install --assume-yes libc6:arm64 -o APT::Immediate-Configure=0 RUN apt-get install --assume-yes libgtk-4-dev:arm64 x11proto-dev:arm64 # RUN apt install --yes libgraphene-1.0-dev:arm64 RUN cargo build --release --target aarch64-unknown-linux-gnu RUN mkdir /result RUN cp target/aarch64-linux-gnu/release/tubefeeder /result RUN echo packaging/tubefeeder.desktop | sed "s/app/usr" > /result/de.schmidhuberj.tubefeeder.desktop RUN cp packaging/tubefeeder.png > /result/de.schmidhuberj.tubefeeder.png RUN echo "Please run `docker cp $(cat /etc/hostname):/result .` to get the results of the compilation". RUN echo "Install `result/tubefeeder` to `/usr/bin/` (or similar, the desktop must be updated if other place)" RUN echo "Install `result/de.schmidhuberj.tubefeeder.desktop` to `/usr/share/applications/` (or similar)" RUN echo "Install `result/de.schmidhuberj.tubefeeder.png` to `/usr/share/icons/hicolor/512x512/apps` (or similar)" RUN echo "TODO: Internationalization" ```

Using a SBC for compilation would likely be very good.