Dowloading ADP video (YouTube) not working any more with VDH7.1.x&CoApp1.1.0 (Gentoo GNU/Linux)

[ALRBP]

Downloading ADP videos from YouTube is not working any more for me. It worked, with some failures, with VDH 7.0.0 and CoApp 1.0.5&1.0.7. With CoApp 1.1.0, it is not working. Tested with VDH 7.1.0, 7.1.1 and 7.1.2a1. I tried re-installing old VDH (7.0.0) & CoApp (1.0.5&1.0.7) and those versions are still working. I am using Gentoo GNU/Linux ~amd64 and Firefox 57.0/57.0.1. I got this bug on two of my computers (all my computers have the same OS), not tested the other two. The error appends quasi-instantly after starting the download. This issue is not present with HLS videos from Dailymotion

Here are the error's details: Exit code: 1 ffprobe version n2.8.13-vdhcoapp Copyright (c) 2007-2017 the FFmpeg developers built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.5) 20160609 configuration: --arch=x86_64 --prefix=/home/mig/git/vdhcoapp/converter/src-build/linux/64/converter-build --extra-version=vdhcoapp --extra-cflags=-I/home/mig/git/vdhcoapp/converter/src-build/linux/64/deps/include --extra-ldflags='-L/home/mig/git/vdhcoapp/converter/src-build/linux/64/deps/lib -L/home/mig/git/vdhcoapp/converter/src-build/linux/64/zlib -static-libgcc' --pkg-config=pkg-config --pkgconfigdir=/home/mig/git/vdhcoapp/converter/src-build/linux/64/deps/lib/pkgconfig --extra-libs=-ldl --enable-shared --enable-gpl --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxvid --enable-libx264 --enable-avresample --disable-indev=sndio --disable-outdev=sndio libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libavresample 2. 1. 0 / 2. 1. 0 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 libpostproc 53. 3.100 / 53. 3.100 /tmp/vdh-321422XX5VBaaPxjm.tmp: Invalid data found when processing input

[mi-g]

Is there any chance your /tmp is on a NTFS or network-mounted volume ? Another user reported a similar issue when using a NTFS drive.

[ALRBP]

My /tmp is on my "/" partition. On my old desktop it is an Ext4 filesystem, Btrfs on my (newer) laptop, local SATA SSD in both cases.

[mi-g]

You wrote:

The error appends quasi-instantly after starting the download

I presume it's not on a very small video right ?

Are you using a proxy ? Have changed anything non-trivial in the add-on settings ?

[ALRBP]

I got issue with all videos I tested, including a 50min/300Mio video and a 3min/10Mio one. I'm using a VPN (OpenVPN) with my own server. I did not changed any VDH setting. For information, I have uBlockOrigin but deactivating it has not effect. I tried deactivating my VPN, it solved the problem for VDH 7.1.1 + CoApp 1.1.0. But VDH 7.0.0 + CoApp 1.0.x is working even with an activated VPN, I just tested, so that still looks like a regression.

[mi-g]

VDH7.0.0 cannot work on some sites and there is no workaround for that. This is due to some restrictions in Firefox 57. This is the reason why 7.1.x implemented the "download from coapp" feature.

I also use OpenVPN on Linux (Ubuntu 16.04) and it works well for me, so we know it's not just a "problem with VPN".

Do we agree you configured your VPN at IP level ? I mean whatever network access you make go through the VPN (except for local network accesses), and not something you configure to say "only Firefox traffic goes through the VPN", if that's even possible.

[ALRBP]

I know the problem with WebExtensions and why the CoApp is now necessary for some kinds of videos that need to be assembled. VDH7.0.0 + CoApp 1.0.x and Firefox 57 work for me on YouTube with my VPN, with some bugs but I am still able to download all videos I tested. I use NetworkManager's OpenVPN plugin for connecting to my VPN. As far as I know, all my traffic is using the VPN. Firefox uses it, the "ping" command uses it (pretty sure because of IPv4/IPv6 considerations). For information, the non-VPN connection I have access to immediately is an IPv4-only connection, while my VPN is an IPv4+IPv6 connection. Tested with ipv6-test.com, the only thing I don't have is a reverse DNS record.

[mi-g]

I know the problem with WebExtensions and why the CoApp is now necessary for some kinds of videos that need to be assembled.

We knew for a while we would need this coapp for file writing. We did not anticipate we would also need it for network requests.

VDH7.0.0 + CoApp 1.0.x and Firefox 57 work for me on YouTube with my VPN, with some bugs but I am still able to download all videos I tested.

Yes, it works in a number of cases, when the server does not verify the Referer nor Cookie headers. But since it cannot work everytime, this is not a viable solution.

Would you be able to use tcpdump or wireshark to monitor the network access made by the coapp ?

[ALRBP]

Never done that before honestly. I only managed to capture all my traffic, but I ensured that no thing else (except Firefox, with only the relevant tab opened) was using network. Still not sure it's safe to make that publicly visible on the Internet. Do you search for something in particular or do you need the whole dump?

[mi-g]

We are looking for the request made by the coapp to the server. The difficulty will be not to mix up with the browser traffic doing the same thing. But we expect the coapp to be different since it does not work.

[mi-g]

Humm, no, this won't work since YouTube is doing HTTPS request, impossible to sniff ...

[ALRBP]

Humm, no, this won't work since YouTube is doing HTTPS request, impossible to sniff ...

Should I upload the whole dump or will this be useless? Anything else I could do?

[mi-g]

The dump will be useless, so no thanks. For now, i don't see how to make progress here. Sorry.

[jonasstein]

@ALRBP which ebuild did you use to install it on Gentoo, or did you compile manually or install a binary? Is your system perhaps on gcc-6.4 and the binary was compiled in gcc-5 and needs external libs?

[ALRBP]

I downloaded it from the official site. It is compiled with GCC 5. On both systems I tested, I have GCC 5 (still required by an old software I have), 6 (sometime necessary if building with 7 fails) and 7 (default) installed. No missing library was reported (the CoApp already reported libnuma missing, solved by installing numactl, an the error output was not the same).

For I am curious to know if the problem is linked to OpenVPN or IPv6. Will find out shortly since I will have access to a non-VPN IPv6 connection later this week.

[ALRBP]

I tested with a non-VPN IPv6 connection and the problem was OpenVPN-related, not IPv6-related.

I say "was" because updating the CoApp to 1.1.1 solved the problem.

Thank you mi-g.

[ALRBP]

OK, ADP WebM videos still fail most of the time (not always), while ADP MP4 are working almost every time. Before 1.1.1, everything was failing every time. ADP WebM works fine with VDH 7.0.0 + CoApp 1.0.7.

I think this should be a good indication of the source of the issue, something that is happening only for ADP WebM in 1.1.1 and was appending also for ADP MP4 in 1.1.0.

[mi-g]

Ok, there has been a lot of strange things were reported leading to corrupted data when downloading with the coapp, and unfortunately i couldn't reproduce the problem, it always works in my test environments. My next move is to rewrite the actual coapp downloads library. It is currently a popular module but does not give much control. Anyway, this will have to wait for the next year.

[mi-g]

There has been a number of fixes on the coapp and add-on and this issue (Invalid data found when processing input) is no longer reported by users. I'm closing the bug.