search

netblue30 / firejail

Linux namespaces and seccomp-bpf sandbox
https://firejail.wordpress.com
GNU General Public License v2.0
5.59k stars 555 forks source link

youtube-dl and ffprobe requires libblas.so #3506

Open hyiltiz opened 4 years ago

hyiltiz commented 4 years ago

Write clear, concise and in textual form.

Bug and expected behavior

No profile or disabling firejail

Reproduce Steps to reproduce the behavior:

  1. Run in bash firejail youtube-dl -x SOMELINK or firejail ffprobe SOMEFILE
  2. See error: 
    
> firejail ffprobe SOMEFILE.webm
Error fcopy: size limit of 500 MB reached
Error getpwuid: main.c:294 init_cfg: No such file or directory

ffprobe SOMEFILE.webm Error fcopy: size limit of 500 MB reached /usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

/usr/bin/ffprobe SOMEFILE.webm ffprobe version 4.3-2 Copyright (c) 2007-2020 the FFmpeg developers built with gcc 9 (Debian 9.3.0-13) ...

Environment

Compile time support:

Checklist

OUTPUT OF `firejail --debug PROGRAM`
https://termbin.com/3iou

BTW, I just noticed that the above firejail --debug provided way too much information about my filesystem and setup. Was that necessary? If so, please consider creating a key pair and uploading your public key so these kinds of probably sensitive private information can be shared directly with the project without putting it up in the Internet indefinitely in plain text.

bbhtt commented 4 years ago

Error fcopy: size limit of 500 MB reached

See here

Error getpwuid: main.c:294 init_cfg: No such file or directory

Can you add passwd,groups,login.defs to private-etc and see if it persists or comment it to see? Is there something wrong with the first line? I ran the command,it works fine:

Output ``` firejail --ignore=quiet youtube-dl --extract-audio --audio-format mp3 https://www.youtube.com/watch?v=vWwgrjjIMXA Reading profile /etc/firejail/youtube-dl.profile Reading profile /etc/firejail/allow-python2.inc Reading profile /etc/firejail/allow-python3.inc Reading profile /etc/firejail/disable-common.inc Reading profile /etc/firejail/disable-devel.inc Reading profile /etc/firejail/disable-exec.inc Reading profile /etc/firejail/disable-interpreters.inc Reading profile /etc/firejail/disable-passwdmgr.inc Reading profile /etc/firejail/disable-programs.inc Reading profile /etc/firejail/disable-xdg.inc Reading profile /etc/firejail/whitelist-usr-share-common.inc Reading profile /etc/firejail/whitelist-var-common.inc Warning: networking feature is disabled in Firejail configuration file Parent pid 8510, child pid 8511 Warning: skipping crypto-policies for private /etc Warning: skipping youtube-dl.conf for private /etc Private /etc installed in 42.92 ms 25 programs installed in 79.52 ms Warning: An abstract unix socket for session D-BUS might still be available. Use --net or remove unix from --protocol set. Blacklist violations are logged to syslog Child process initialized in 236.85 ms [youtube] vWwgrjjIMXA: Downloading webpage [download] Destination: Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.webm [download] 100% of 2.45MiB in 00:03 [ffmpeg] Destination: Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.mp3 Deleting original file Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.webm (pass -k to keep) Parent is shutting down, bye... ```

/usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

ldconfig -p | grep -i "libblas" shows anything?

hyiltiz commented 4 years ago

Error fcopy: size limit of 500 MB reached

See here Seems it is compiled into firejail; not much I can do unless recompile myself... It would be great to be as a config parameter though.

Error getpwuid: main.c:294 init_cfg: No such file or directory

Can you add passwd,groups,login.defs to private-etc and see if it persists or comment it to see? Is there something wrong with the first line? I ran the command,it works fine:

$ firejail --private-etc=passwd,groups,login.defs, ffprobe SOMEFILE.webm
Error fcopy: size limit of 500 MB reached
/usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

Output

/usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

ldconfig -p | grep -i "libblas" shows anything?

$ ldconfig -p | grep -i "libblas"
libblas.so.3 (libc6,x86-64) => /lib/x86_64-linux-gnu/libblas.so.3
libblas.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libblas.so
glitsj16 commented 4 years ago

Error fcopy: size limit of 500 MB reached

This can be set in /etc/firejail/firejail.config:

[...]
# Set the limit for file copy in several --private-* options. The size is set
# in megabytes. By default we allow up to 500MB.
# Note: the files are copied in RAM.
# file-copy-limit 500
[...]

Can you post output of

$ firejail --private-etc=ld.so.cache /usr/bin/ffprobe https://www.youtube.com/watch?v=vWwgrjjIMXA
bbhtt commented 4 years ago

$ firejail --private-etc=ld.so.cache /usr/bin/ffprobe https://www.youtube.com/watch?v=vWwgrjjIMXA

ffprobe cannot parse direct web links,you need a direct link to the video,not the https site,I think. https://ia800705.us.archive.org/13/items/Jolly_Fish_1932/Tom_and_Jerry_Jolly_Fish_1932_512kb.mp4 I couldn't reproduce this;ffmpeg profiles needed to be changed: ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload from current in git

hyiltiz commented 4 years ago 
hyiltiz ~ $ firejail --private-etc=ld.so.cache /usr/bin/youtube-dl https://www.youtube.com/watch?v=vWwgrjjIMXA
Error fcopy: size limit of 500 MB reached
[youtube] vWwgrjjIMXA: Downloading webpage
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.f136.mp4
[download] 100% of 7.57MiB in 00:00
[download] Destination: Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.f251.webm
[download] 100% of 2.45MiB in 00:00
[ffmpeg] Merging formats into "Blowing In The Wind (Live On TV, March 1963)-vWwgrjjIMXA.mkv"
ERROR: ffmpeg: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory
[2]+  Done                    zotero

hyiltiz ~ $ firejail --private-etc=ld.so.cache /usr/bin/ffprobe Blowing\ In\ The\ Wind\ \(Live\ On\ TV\,\ March\ 1963\)-vWwgrjjIMXA.f251.webm
Error fcopy: size limit of 500 MB reached
/usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

hyiltiz ~ $ firejail --private-etc=ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload /usr/bin/ffprobe Blowing\ In\ The\ Wind\ \(Live\ On\ TV\,\ March\ 1963\)-vWwgrjjIMXA.f251.webm
Error fcopy: size limit of 500 MB reached
/usr/bin/ffprobe: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory
rusty-snake commented 4 years ago

@hyiltiz has you made any modifications to the ffprobe, ffmpeg, youtube-dl profiles?

Error getpwuid: main.c:294 init_cfg: No such file or directory

@kortewegdevries that happens if you do firejail APP where APP has a symlink from firecfg.

hyiltiz commented 4 years ago

@rusty-snake I don't recall any modifications. My profiles:

$ cat /etc/firejail/ffmpeg.profile /etc/firejail/ffprobe.profile /etc/firejail/youtube-dl.profile | nc termbin.com 9999
https://termbin.com/eff2
rusty-snake commented 4 years ago

Looks then like you need to try which line is the issue. Copy the ffmpeg profile to your home and comment all line. Then uncomment line for line. Use firejail --profile=ffmpeg.profile ffprobe … to start. Note1: private-bin should commented all the time. Note2: My feeling is private-etc but I'm not sure.

hyiltiz commented 4 years ago

Tested with firejail --profile=ffmpeg.profile ffprobe ~/Downloads/media/SOMEFILE.webm after copying /etc/firejail/ffmpeg.profile to current directory ~/.config/firejail.

Changing nothing:

> firejail --profile=ffmpeg.profile ffprobe ~/Downloads/media/SOMEFILE.webm
Error fcopy: size limit of 500 MB reached
Error: no suitable ffprobe executable found

Commenting out private-bin:

> firejail --profile=ffmpeg.profile ffprobe ~/Downloads/media/SOMEFILE.webm
Error fcopy: size limit of 500 MB reached
Error getpwuid: main.c:294 init_cfg: No such file or directory

Also commenting out private-etc worked!

> firejail --profile=ffmpeg.profile ffprobe ~/Downloads/media/SOMEFILE.webm
ffprobe version 4.3-3 Copyright (c) 2007-2020 the FFmpeg developers
...
    encoder         : google/video-file
  Duration: 00:41:43.18, start: -0.007000, bitrate: 138 kb/s
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default)

Any idea why private-bin and private-etc are uncommented in master?

rusty-snake commented 4 years ago

private-bin: the ffprobe.profile contains ignore private-bin private-etc: try to use the private-etc from master, it has some fixes for dynamic linked librarys.

hyiltiz commented 4 years ago

Can I just put the master private-etc under ~/.config/firejail/?

rusty-snake commented 4 years ago

Yes you can add private-etc ... to .config/firejail/ffmpeg.local. Alternative you can also edit /etc/firejail/ffmpeg.profile since the next firejail release will fix it anyway.

ytdl needs ffprobe added to its private-bin (also on master).

hyiltiz commented 4 years ago

Got the private-etc ... line from master

private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl

to /etc/firejail/ffmpeg.profile, and added ffprobe to /etc/firejail/youtube-dl.profile's private-bin line before youtube-dl like:

private-bin env,ffmpeg,ffprobe,python*,youtube-dl

Still getting (other) errors about ffprobe:

> youtube-dl -x 'https://www.youtube.com/watch?v=abcde'
Error fcopy: size limit of 500 MB reached
[youtube] abcde: Downloading webpage
[download] SOMEFILE.webm has already been downloaded
[download] 100% of 41.22MiB
ERROR: WARNING: unable to obtain file audio codec with ffprobe
> ffprobe SOMEFILE.webm
ffprobe version 4.3-3 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Debian 9.3.0-14)
...
Input #0, matroska,webm, from 'SOMEFILE.webm':
  Metadata:
    encoder         : google/video-file
  Duration: 00:41:43.18, start: -0.007000, bitrate: 138 kb/s
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
rusty-snake commented 4 years ago

I just saw that the private-etc line in youtube-dl.profile also needs to extended with ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload.

hyiltiz commented 4 years ago

Added, but still same error ERROR: WARNING: unable to obtain file audio codec with ffprobe

rusty-snake commented 4 years ago 
$ fjp diff ffmpeg youtube-dl
The following options are unique to ffmpeg.profile:
include ffmpeg.local
whitelist /usr/share/devedeng
whitelist /usr/share/ffmpeg
whitelist /usr/share/qtchooser
protocol protocol inet,inet6

seccomp !set_mempolicy
private-bin ffmpeg
private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl

The following options are unique to youtube-dl.profile:
include youtube-dl.local
ignore noexec ${HOME}
noblacklist ${HOME}/.cache/youtube-dl
noblacklist ${HOME}/.config/youtube-dl
noblacklist ${HOME}/.netrc
include allow-python2.inc
include allow-python3.inc
blacklist /tmp/.X11-unix
blacklist ${RUNUSER}/wayland-*
blacklist ${RUNUSER}
no3d
protocol protocol unix,inet,inet6

seccomp
private-bin env,ffmpeg,python*,youtube-dl
private-etc alternatives,ca-certificates,crypto-policies,hostname,hosts,ld.so.cache,mime.types,pki,resolv.conf,ssl,youtube-dl.conf

Lets go trough it. whitelist /usr/share/ffmpeg looks most promising to me right now, try to add it to ytdl.

hyiltiz commented 4 years ago

Same error; even tried appending the diff'ed lines:

include ffmpeg.local
whitelist /usr/share/devedeng
whitelist /usr/share/ffmpeg
whitelist /usr/share/qtchooser
protocol protocol inet,inet6

seccomp !set_mempolicy
private-bin ffmpeg
private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl

to /etc/firejail/youtube-dl, but got the same error.

bbhtt commented 4 years ago

Can you see if --prefer-ffmpeg works? Or attach verbose and ignore-config flags to youtube-dl and post the output in a text file.

hyiltiz commented 4 years ago

Here you go (these still keep all the changes we've already done earlier):

> youtube-dl --prefer-ffmpeg -x 'https://www.youtube.com/watch?v=abcd'
Error fcopy: size limit of 500 MB reached
Error fcopy: size limit of 500 MB reached
[youtube] SOMEFILE: Downloading webpage
[download] SOMEFILE.webm has already been downloaded
[download] 100% of 41.22MiB
ERROR: WARNING: unable to obtain file audio codec with ffprobe

> youtube-dl --prefer-ffmpeg --verbose --ignore-config -x 'https://www.youtube.com/watch?v=abcd'
Error fcopy: size limit of 500 MB reached
Error fcopy: size limit of 500 MB reached
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--prefer-ffmpeg', '--verbose', '--ignore-config', '-x', 'https://www.youtube.com/watch?v=abcd']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.06.16.1
[debug] Python version 3.8.4rc1 (CPython) - Linux-5.7.0-1-amd64-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg present, ffprobe present
[debug] Proxy map: {}
[youtube] abcd: Downloading webpage
[debug] Invoking downloader on 'https:/SOMELINK'
[download] SOMEFILE.webm has already been downloaded
[download] 100% of 41.22MiB
[debug] ffmpeg command line: ffprobe -show_streams 'file:SOMEFILE.webm'
ERROR: WARNING: unable to obtain file audio codec with ffprobe
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 2065, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/lib/python3/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 274, in run
    raise PostProcessingError('WARNING: unable to obtain file audio codec with ffprobe')
youtube_dl.utils.PostProcessingError: WARNING: unable to obtain file audio codec with ffprobe
bbhtt commented 4 years ago

Just a guess, can you try with a shorter video (10-20 MB) or a different file format like mp4,mkv,m4a; also with specifying the format --audio-format? I couldn't reproduce any of it on Debian 10

hyiltiz commented 4 years ago

Here, a 1.14MiB video, .m4a format, specifying --audio-format opus:

$ youtube-dl -x 'https://www.youtube.com/watch?v=7RMz7tePA98'
Error fcopy: size limit of 500 MB reached
Error fcopy: size limit of 500 MB reached
[youtube] 7RMz7tePA98: Downloading webpage
[youtube] 7RMz7tePA98: Downloading js player 5253ac4d
[youtube] 7RMz7tePA98: Downloading js player 5253ac4d
[youtube] 7RMz7tePA98: Downloading MPD manifest
[dashsegments] Total fragments: 9
[download] Destination: Firejail Introduction-7RMz7tePA98.m4a
[download] 100% of 1.14MiB in 00:02
[ffmpeg] Correcting container in "Firejail Introduction-7RMz7tePA98.m4a"
ERROR: ffmpeg: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory

$ youtube-dl -x --audio-format opus 'https://www.youtube.com/watch?v=7RMz7tePA98'
Error fcopy: size limit of 500 MB reached
Error fcopy: size limit of 500 MB reached
[youtube] 7RMz7tePA98: Downloading webpage
[youtube] 7RMz7tePA98: Downloading js player 5253ac4d
[youtube] 7RMz7tePA98: Downloading js player 5253ac4d
[youtube] 7RMz7tePA98: Downloading MPD manifest
[dashsegments] Total fragments: 9
[download] Destination: Firejail Introduction-7RMz7tePA98.m4a
[download] 100% of 1.14MiB in 00:01
[ffmpeg] Correcting container in "Firejail Introduction-7RMz7tePA98.m4a"
ERROR: ffmpeg: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory
bbhtt commented 3 years ago

@hyiltiz Trying this one more time. Can you verify if you have the following files?

/etc/alternatives/libblas.so.3-x86_64-linux-gnu (This is a symlink to 3)
/usr/lib/x86_64-linux-gnu/libblas.so.3  (This one's done!)
/usr/lib/x86_64-linux-gnu/blas/libblas.so.3 (3)
/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
/var/lib/dpkg/alternatives/libblas.so.3-x86_64-linux-gnu

ffprobe needs the library: openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libblas.so.3", O_RDONLY|O_CLOEXEC) = 3

hyiltiz commented 3 years ago 
root@iPhone:~# ls -la /etc/alternatives/libblas.so*
lrwxrwxrwx 1 root root 55 Nov  3  2019 /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
lrwxrwxrwx 1 root root 41 Jan  6  2020 /etc/alternatives/libblas.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/blas/libblas.so

root@iPhone:~# ls -la /usr/lib/x86_64-linux-gnu/libblas.so*
lrwxrwxrwx 1 root root 45 Jan  6  2020 /usr/lib/x86_64-linux-gnu/libblas.so -> /etc/alternatives/libblas.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 47 Mar  9  2019 /usr/lib/x86_64-linux-gnu/libblas.so.3 -> /etc/alternatives/libblas.so.3-x86_64-linux-gnu

root@iPhone:~# ls -la /usr/lib/x86_64-linux-gnu/blas/libblas.so.*
lrwxrwxrwx 1 root root     16 Apr  2 05:56 /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 -> libblas.so.3.9.0
-rw-r--r-- 1 root root 440160 Apr  2 05:56 /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0

root@iPhone:~# ls -la /var/lib/dpkg/alternatives/libblas.so*
-rw-r--r-- 1 root root 201 Aug  2 06:43 /var/lib/dpkg/alternatives/libblas.so.3-x86_64-linux-gnu
-rw-r--r-- 1 root root 421 Apr  6 12:44 /var/lib/dpkg/alternatives/libblas.so-x86_64-linux-gnu
bbhtt commented 3 years ago

Hm, I was under the impression that the symlink under /etc/alternatives/ wasn't pointing to the correct library in /lib/, because I could reproduce that error by installing multiple packages (amd64) that offered the same library. Each time I installed one update-alternatives reconfigured that symlink under /etc/and ffprobe searches for the library under /etc/alternatives/. A fresh Debian Sid comes with only libblas3 pre-installed. This was the stock symlink that worked: /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/blas/libblas.so.3

hyiltiz commented 3 years ago

I have libblas3 and libblas-dev installed; I get rid of libblas-dev? Not sure what you are suggesting.

bbhtt commented 3 years ago

I have libblas3 and libblas-dev installed; I get rid of libblas-dev? Not sure what you are suggesting.

I'm not to sure about it. I guess it's worth a try at this point. You might also have libopenblas0-pthread. So get rid of all 3+ffmpeg (apt-mark hold/unhold), then reinstall only libblas3,ffmpeg,reset the cache sudo ldconfig -N -X -v,try running with the profile that came with 0.9.62 without any modifications first, then add private-etc of master if you need.I don't know what's wrong with you system but I grabbed a copy of Sid NetInst iso (-344 Mb) took 30 minutes to install and setup firejail and I can run fine with 0.9.62-3 and the profiles that came with it w/o any modifications. The libblas dependency came in with Sid, you can apt-rdepends ffmpeg to find it's listed under libalapack and libsphinx.

hyiltiz commented 3 years ago

Hmm, I'd be willing to try the above suggestion, but I think it is a workaround rather than a fix if I understand Firejail's "no fuss" design principle correctly. It is better implemented as a rule against various system states rather than fine tuning system packages.

libblas-dev or libblas3* cannot be removed as it is a dependency for a few other packages I'd rather keep. Removed libopenblas0-pthread, libopenblas0* and libopenblas0-openmp*. Removed then installed ffmpeg, then:

~ # ldconfig -N -X -v|grep blas
        libf77blas.so.3 -> libf77blas.so.3.10.3
        libgslcblas.so.0 -> libgslcblas.so.0.0.0
        libcblas.so.3 -> libcblas.so.3.10.3

Even with the private-etc modifications, still getting the same error about libblas.

bbhtt commented 3 years ago

libblas-dev or libblas3* cannot be removed as it is a dependency for a few other packages I'd rather keep.

apt install --reinstall libblas3

hyiltiz commented 3 years ago

Reinstalled libblas3 then ffmpeg hten reset ldconfig cache. Still the same error.

bbhtt commented 3 years ago

Commenting private-etc still works? Like you said:

Also commenting out private-etc worked!

hyiltiz commented 3 years ago

Hmm, I re-downloaded the ffmpeg.profile from master. Had to comment out private-bin ffmpeg, private-etc, and ADDITIONALLY also include disable-shell.inc, dbus-user none and dbus-system none for firejail --profile=~/.config/firejail/ffmpeg.profile ffprobe ~/Downloads/media/SOMEFILE.webm to work.

Uncommenting private-etc ... for the above command says:

Error fcopy: size limit of 500 MB reached
Error getpwuid: main.c:294 init_cfg: No such file or directory
hyiltiz@iPhone /tmp> locate libblas.so
/etc/alternatives/libblas.so-x86_64-linux-gnu
/etc/alternatives/libblas.so.3-x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libblas.so
/usr/lib/x86_64-linux-gnu/libblas.so.3
/usr/lib/x86_64-linux-gnu/atlas/libblas.so.3
/usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
/usr/lib/x86_64-linux-gnu/blas/libblas.so
/usr/lib/x86_64-linux-gnu/blas/libblas.so.3
/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
/var/lib/dpkg/alternatives/libblas.so-x86_64-linux-gnu
/var/lib/dpkg/alternatives/libblas.so.3-x86_64-linux-gnu
bbhtt commented 3 years ago

@kortewegdevries that happens if you do firejail APP where APP has a symlink from firecfg.

firejail --profile=<> /usr/bin/<> . Those were not in 0.9.62.

hyiltiz commented 3 years ago

firejail --profile=<> /usr/bin/<> . Those were not in 0.9.62.

Not sure about your <> notation.

bbhtt commented 3 years ago

Not sure about your <> notation.

That error happens when you already have a symlink. firejail --profile=youtube-dl /usr/bin/youtube-dl.

hyiltiz commented 3 years ago

Not sure the last command was what you'd like me to test, but here is what I did:

  1. Downloaded the ffmpeg.profile and youtube-dl.profile from master
  2. Commented out private-bin, private-etc, include disable-shell.inc, dbus-user, dbus-system lines
  3. Ran firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'

So far, it is the only setup that works reliably.

bbhtt commented 3 years ago

So the problematic line is either private-etc or private-bin? dbus-user dbus-system aren't available to 0.9.62. Can you try narrowing down it a bit to whose private-etc or private-bin is causing the problem?. Like for example, test with only ffmpeg somefile.mp4 with your profile to see:   1. if the master works without any change 

  1. commenting private-bin or private-etc helps, then move on to the youtube-dl profile...

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐On Sunday, August 16, 2020 4:44 AM, Hörmet Yiltiz notifications@github.com wrote:

Not sure the last command was what you'd like me to test, but here is what I did:

  1. Downloaded the ffmpeg.profile and youtube-dl.profile from master
  2. Commented out private-bin, private-etc, include disable-shell.inc, dbus-user, dbus-system lines
  3. Ran firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'

So far, it is the only setup that works reliably.

—

You are receiving this because you were mentioned.

hyiltiz commented 3 years ago
  1. The master doesn't work without any change;
  2. dbus-* etc. needs to be removed for 0.9.62 for firejail to parse first so did
  3. only commenting out private-bin fails
  4. only commenting out private-etc succeeds

For detailed procedures, here is what I did (script typescript output, can be viewed as plain-text file) http://ix.io/2ujM

bbhtt commented 3 years ago

only commenting out private-etc succeeds

This is for ffmpeg or youtube-dl or both? Sorry, I can't open your link...

hyiltiz commented 3 years ago

Both. As you asked, I tested for ffmpeg/ffprobe only first, then with a successful ffmpeg setup, tested youtube-dl.

bbhtt commented 3 years ago

Try adding this to the private-etc line of youtube-dl.profile separated by commas, remove if one is duplicate, "ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf" to see if it works: firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'. You can remove the quiet from ffprobe,ffmpeg,ytdl profile to get a better overview of which profiles are being loaded by firejail...

If this still fails, try a combination of strace (strace -o file.txt /usr/bin/ytdl <link> and cat file.txt|egrep "/etc/|/usr/share/" or under firejail: firejail --allow-debuggers --profile=profile strace -o file.txt /usr/bin/ytdl <link>) and firejail --build /usr/bin/ytdl( should output info on terminal,if not look in /tmp for a file named firejail_straceXXX`) to see if something comes up specific to your system that isn't in any of the profiles.

hyiltiz commented 3 years ago

Try adding this to the private-etc line of youtube-dl.profile separated by commas, remove if one is duplicate, "ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf"

Did that.

 $ cat ~/.config/firejail/youtube-dl.profile 
# Firejail profile for youtube-dl
# Description: Downloader of videos from YouTube and other sites
# This file is overwritten after every install/update
quiet
# Persistent local customizations
include youtube-dl.local
# Persistent global definitions
include globals.local

# breaks when installed under ${HOME} via `pip install --user` (see #2833)
ignore noexec ${HOME}

noblacklist ${HOME}/.cache/youtube-dl
noblacklist ${HOME}/.config/youtube-dl
noblacklist ${HOME}/.netrc
noblacklist ${MUSIC}
noblacklist ${VIDEOS}

# Allow python (blacklisted by disable-interpreters.inc)
include allow-python2.inc
include allow-python3.inc

blacklist /tmp/.X11-unix
blacklist ${RUNUSER}/wayland-*
blacklist ${RUNUSER}

include disable-common.inc
include disable-devel.inc
include disable-exec.inc
include disable-interpreters.inc
include disable-passwdmgr.inc
include disable-programs.inc
# include disable-shell.inc
include disable-xdg.inc

include whitelist-usr-share-common.inc
include whitelist-var-common.inc

apparmor
caps.drop all
ipc-namespace
machine-id
netfilter
no3d
nodvd
nogroups
nonewprivs
noroot
nosound
notv
nou2f
novideo
protocol unix,inet,inet6
seccomp
shell none
tracelog

private-bin env,ffmpeg,python*,youtube-dl
private-cache
private-dev
# private-etc alternatives,ca-certificates,crypto-policies,hostname,hosts,ld.so.cache,mime.types,pki,resolv.conf,ssl,youtube-dl.conf
private-etc ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf
private-tmp

# dbus-user none
# dbus-system none

#memory-deny-write-execute - breaks on Arch (see issue #1803)
hyiltiz@iPhone ~ $ cat ~/.config/firejail/ffmpeg.profile
# Firejail profile for ffmpeg
# Description: Tools for transcoding, streaming and playing of multimedia files
# This file is overwritten after every install/update
quiet
# Persistent local customizations
include ffmpeg.local
# Persistent global definitions
include globals.local

noblacklist ${MUSIC}
noblacklist ${VIDEOS}

include disable-common.inc
include disable-devel.inc
include disable-exec.inc
include disable-interpreters.inc
include disable-passwdmgr.inc
include disable-programs.inc
# include disable-shell.inc
include disable-xdg.inc

whitelist /usr/share/devedeng
whitelist /usr/share/ffmpeg
whitelist /usr/share/qtchooser
include whitelist-usr-share-common.inc
include whitelist-var-common.inc

apparmor
caps.drop all
ipc-namespace
machine-id
netfilter
nodvd
nogroups
nonewprivs
noroot
nosound
notv
nou2f
novideo
protocol inet,inet6
# allow set_mempolicy, which is required to encode using libx265
seccomp !set_mempolicy
shell none
tracelog

private-bin ffmpeg
private-cache
private-dev
# private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl
private-tmp

# dbus-user none
# dbus-system none

# memory-deny-write-execute - it breaks old versions of ffmpeg

to see if it works: firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'. Tested, which failed with a new error.

firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
[youtube] 7RMz7tePA98: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno -3] Temporary failure in name resolution> (caused by URLError(gaierror(-3, 'Temporary failure in name resolution')))

You can remove the quiet from ffprobe,ffmpeg,ytdl profile to get a better overview of which profiles are being loaded by firejail...

Removed quiet.

hyiltiz@iPhone ~ $ firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Reading profile /home/hyiltiz/.config/firejail/youtube-dl.profile
Reading profile /etc/firejail/allow-python2.inc
Reading profile /etc/firejail/allow-python3.inc
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-devel.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 159161, child pid 159162
Warning: skipping httpd for private /etc
Warning: skipping apache for private /etc
Private /etc installed in 15.64 ms
41 programs installed in 72.33 ms
Blacklist violations are logged to syslog
Child process initialized in 186.68 ms
[youtube] 7RMz7tePA98: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno -3] Temporary failure in name resolution> (caused by URLError(gaierror(-3, 'Temporary failure in name resolution')))

Parent is shutting down, bye...
hyiltiz@iPhone ~ :( $ firejail --profile=NONE /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'Error: no profile with name "NONE" found.
hyiltiz@iPhone ~ :( $ firejail --profile=/dev/null /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'

If this still fails, try a combination of strace (strace -o file.txt /usr/bin/ytdl <link> and cat file.txt|egrep "/etc/|/usr/share/"

Failed, so trying strace. 

hyiltiz@iPhone ~ $ strace -o dump.txt /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
[youtube] 7RMz7tePA98: Downloading webpage
[youtube] 7RMz7tePA98: Downloading MPD manifest
[dashsegments] Total fragments: 15
[download] Destination: Firejail Introduction-7RMz7tePA98.f136.mp4
[download] 100% of 4.36MiB in 00:03
[dashsegments] Total fragments: 9
[download] Destination: Firejail Introduction-7RMz7tePA98.f140.m4a
[download] 100% of 1.14MiB in 00:01
[ffmpeg] Merging formats into "Firejail Introduction-7RMz7tePA98.mp4"
Deleting original file Firejail Introduction-7RMz7tePA98.f136.mp4 (pass -k to keep)
Deleting original file Firejail Introduction-7RMz7tePA98.f140.m4a (pass -k to keep)
hyiltiz@iPhone ~ $ rm Firejail\ Introduction-7RMz7tePA98.mp4 
hyiltiz@iPhone ~ $ cat dump.txt|egrep "/etc/|/usr/share/"
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcb310) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/mime.types", {st_mode=S_IFREG|0644, st_size=24546, ...}) = 0
openat(AT_FDCWD, "/etc/mime.types", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/httpd/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/etc/httpd/conf/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/etc/apache/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/etc/apache2/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/usr/local/etc/httpd/conf/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/usr/local/etc/httpd/conf/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
stat("/usr/local/etc/mime.types", 0x7fffa4dc88a0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd490) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1e0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en_US/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale-langpack/en/LC_MESSAGES/messages.mo", 0x7fffa4dcd1a0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/youtube-dl.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
read(3, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 494
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3536, ...}) = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4

or under firejail: firejail --allow-debuggers --profile=profile strace -o file.txt /usr/bin/ytdl <link>)

This failed since strace wasn't found in the youtube-dl profile. Didn't want to further fuss with the profile we are trying to debug in the first place. But I'll do so if that is necessary.

hyiltiz@iPhone ~ $ firejail --allow-debuggers --profile=~/.config/firejail/youtube-dl.profile strace -o dump2.txt /usr/bin/ytdl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Reading profile /home/hyiltiz/.config/firejail/youtube-dl.profile
Reading profile /etc/firejail/allow-python2.inc
Reading profile /etc/firejail/allow-python3.inc
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 160036, child pid 160037
Warning: skipping httpd for private /etc
Warning: skipping apache for private /etc
Private /etc installed in 20.98 ms
41 programs installed in 72.22 ms
Blacklist violations are logged to syslog
Child process initialized in 192.22 ms
Error: no suitable strace executable found

Parent is shutting down, bye...
hyiltiz@iPhone ~ :( $ which strace
/usr/bin/strace
hyiltiz@iPhone ~ $ firejail --allow-debuggers --profile=~/.config/firejail/youtube-dl.profile /usr/bin/strace -o dump2.txt /usr/bin/ytdl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Reading profile /home/hyiltiz/.config/firejail/youtube-dl.profile
Reading profile /etc/firejail/allow-python2.inc
Reading profile /etc/firejail/allow-python3.inc
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 160094, child pid 160095
Warning: skipping httpd for private /etc
Warning: skipping apache for private /etc
Private /etc installed in 15.13 ms
41 programs installed in 63.22 ms
Blacklist violations are logged to syslog
Child process initialized in 179.90 ms
Error: no suitable /usr/bin/strace executable found

Parent is shutting down, bye...
hyiltiz@iPhone ~ :( $ firejail --allow-debuggers --profile=profile /usr/bin/strace -o dump2.txt /usr/bin/ytdl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Error: no profile with name "profile" found.
hyiltiz@iPhone ~ :( $ firejail --allow-debuggers --profile=profile strace -o dump2.txt /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Error: no profile with name "profile" found.
hyiltiz@iPhone ~ :( $ firejail --allow-debuggers --profile=~/.config/firejail/youtube-dl.profile   strace -o dump2.txt /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Reading profile /home/hyiltiz/.config/firejail/youtube-dl.profile
Reading profile /etc/firejail/allow-python2.inc
Reading profile /etc/firejail/allow-python3.inc
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 160162, child pid 160163
Warning: skipping httpd for private /etc
Warning: skipping apache for private /etc
Private /etc installed in 16.70 ms
41 programs installed in 71.31 ms
Blacklist violations are logged to syslog
Child process initialized in 185.52 ms
Error: no suitable strace executable found

Parent is shutting down, bye...

and firejail --build /usr/bin/ytdl( should output info on terminal,if not look in /tmp for a file named firejail_straceXXX`) to see if something comes up specific to your system that isn't in any of the profiles.

This does seem to show interesting results but I am not sure what's specific to my system. It mentioned seccomp in the end, so provided a few filters for the mentioned logs. I'd also paste the strace dump but it is 20k LOC, and even pastebin is refusing to handle that.

$ firejail --build /usr/bin/youtube-dl  'https://www.youtube.com/watch?v=7RMz7tePA98'
[youtube] 7RMz7tePA98: Downloading webpage
[youtube] 7RMz7tePA98: Downloading MPD manifest
[dashsegments] Total fragments: 15
[download] Destination: Firejail Introduction-7RMz7tePA98.f136.mp4
[download] 100% of 4.36MiB in 00:02
[dashsegments] Total fragments: 9
[download] Destination: Firejail Introduction-7RMz7tePA98.f140.m4a
[download] 100% of 1.14MiB in 00:04
[ffmpeg] Merging formats into "Firejail Introduction-7RMz7tePA98.mp4"
Deleting original file Firejail Introduction-7RMz7tePA98.f136.mp4 (pass -k to keep)
Deleting original file Firejail Introduction-7RMz7tePA98.f140.m4a (pass -k to keep)
--- Built profile beings after this line ---
############################################
# /usr/bin/youtube-dl profile
############################################
# Persistent global definitions
# include /etc/firejail/globals.local

### basic blacklisting
include /etc/firejail/disable-common.inc
# include /etc/firejail/disable-devel.inc
include /etc/firejail/disable-passwdmgr.inc
# include /etc/firejail/disable-programs.inc

### home directory whitelisting
whitelist ~/youtube-dl.conf.txt
whitelist ~/youtube-dl.conf
include /etc/firejail/whitelist-common.inc

### filesystem

# private-tmp
# File accessed in /tmp directory:
# /tmp/firejail-strace.wHKbFJ,
private-dev
private-etc gcrypt,selinux,login.defs,passwd,hosts,youtube-dl.conf,mime.types,
blacklist /var
private-bin ffprobe,ffmpeg,bash,firejail,uname,python3.8,
# private-lib
blacklist /usr/share

### security filters
caps.drop all
nonewprivs
seccomp
# seccomp.keep wait4,futex,read,mmap,write,openat,stat,close,poll,fstat,mprotect,ioctl,getdents64,execve,lseek,unlink,connect,clone,socket,getsockopt,munmap,sendmmsg,rename,recvmsg,recvfrom,setsockopt,getsockname,getpid,sendto,brk,rt_sigaction,access,lstat,utimensat,setresuid,setresgid,getuid,readlink,bind,getpeername,rt_sigprocmask,geteuid,getgid,pipe2,umask,prlimit64,arch_prctl,set_robust_list,getcwd,select,sysinfo,uname,set_tid_address,statfs,getegid,getppid,getpgrp,mlock,sched_getaffinity,fcntl,getrandom,prctl,getrusage,gettid,madvise,dup,dup2,sigaltstack
# 68 syscalls total
# Probably you will need to add more syscalls to seccomp.keep. Look for
# seccomp errors in /var/log/syslog or /var/log/audit/audit.log while
# running your sandbox.

### network
protocol inet,inet6,
# net eth0
netfilter

### environment
shell none
hyiltiz@iPhone ~ $ ^C
hyiltiz@iPhone ~ :( $ rg seccomp /var/log/syslog 
414:Sep 21 14:28:51 iPhone kernel: [    0.107616] Spectre V2 : User space: Mitigation: STIBP via seccomp and prctl
416:Sep 21 14:28:51 iPhone kernel: [    0.107617] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
1506:Sep 21 14:28:53 iPhone chronyd[1703]: Loaded seccomp filter
hyiltiz@iPhone ~ $ rg seccomp /var/log/audit/audit.log
hyiltiz@iPhone ~ :( $ exit
rusty-snake commented 3 years ago

What the state here?

hyiltiz commented 3 years ago

I've been getting help trying to pinpoint a profile that works and why it wasn't working. Still awaiting help.

bbhtt commented 3 years ago

Try adding this to the private-etc line of youtube-dl.profile separated by commas, remove if one is duplicate, "ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf"

Did that.

Sorry if there was a confusion but you should've added it to the original private-etc line not make a new line like this:

# private-etc alternatives,ca-certificates,crypto-policies,hostname,hosts,ld.so.cache,mime.types,pki,resolv.conf,ssl,youtube-dl.conf private-etc ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf private-tmp

So only one private-etc like this:

private-etc alternatives,apache,apache2,ca-certificates,crypto-policies,gai.conf,hostname,hosts,host.conf,httpd,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,mime.types,nsswitch.conf,pki,resolv.conf,ssl,youtube-dl.conf

Anyways looking at the strace and build I don't think even above will work and in that case I'm out of ideas; sorry.

hyiltiz commented 3 years ago

Huh? There was no extra newlines in my tests in private-etc line; the first line is commented out with #, and the second line, IIRC, should be one I copied out from this thread.

Shall we keep this open to see if someone else may have some ideas? @kortewegdevries couldn't replicate it, so maybe waiting until someone else could replicate would be meaningful too.

bbhtt commented 3 years ago

Huh? There was no extra newlines in my tests in private-etc line; the first line is commented out with #, and the second line, IIRC, should be one I copied out from this thread.

You commented the original private-etc line from the profile and added only this private-etc ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf.

Hence the error in your post: (because with only that you are missing hosts,resolv.conf, etc.)

ERROR: Unable to download webpage: <urlopen error [Errno -3] Temporary failure in name resolution> (caused by URLError(gaierror(-3, 'Temporary failure in name resolution')))

I said to combine both like this: private-etc alternatives,apache,apache2,ca-certificates,crypto-policies,gai.conf,hostname,hosts,host.conf,httpd,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,mime.types,nsswitch.conf,pki,resolv.conf,ssl,youtube-dl.conf not comment the original one and add another line.

hyiltiz commented 3 years ago

Aha, gotcha! Thanks for patiently explaining. Did with only changing youtube-dl profile and the ffmpeg profile, and then also (the system-wide):

yiltiz@iPhone ~/Downloads [1]> grep private-etc ~/.config/firejail/ffmpeg.profile 
# private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl
private-etc alternatives,apache,apache2,ca-certificates,crypto-policies,gai.conf,hostname,hosts,host.conf,httpd,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,mime.types,nsswitch.conf,pki,resolv.conf,ssl,youtube-dl.conf
hyiltiz@iPhone ~/Downloads> grep private-etc ~/.config/firejail/youtube-dl.profile 
# private-etc alternatives,ca-certificates,crypto-policies,hostname,hosts,ld.so.cache,mime.types,pki,resolv.conf,ssl,youtube-dl.conf
# private-etc ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,httpd,apache,apache2,nsswitch.conf,host.conf,gai.conf
private-etc alternatives,apache,apache2,ca-certificates,crypto-policies,gai.conf,hostname,hosts,host.conf,httpd,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,mime.types,nsswitch.conf,pki,resolv.conf,ssl,youtube-dl.conf
hyiltiz@iPhone ~/Downloads> rm Firejail\ Introduction-7RMz7tePA98.f1*
hyiltiz@iPhone ~/Downloads> grep private-etc /etc/firejail/ffmpeg.profile
# private-etc alternatives,ca-certificates,crypto-policies,hosts,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,nsswitch.conf,pkcs11,pki,resolv.conf,ssl
private-etc alternatives,apache,apache2,ca-certificates,crypto-policies,gai.conf,hostname,hosts,host.conf,httpd,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,localtime,mime.types,nsswitch.conf,pki,resolv.conf,ssl,youtube-dl.conf
hyiltiz@iPhone ~/Downloads> firejail --profile=~/.config/firejail/youtube-dl.profile /usr/bin/youtube-dl 'https://www.youtube.com/watch?v=7RMz7tePA98'
Reading profile /home/hyiltiz/.config/firejail/youtube-dl.profile
Reading profile /etc/firejail/allow-python2.inc
Reading profile /etc/firejail/allow-python3.inc
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-devel.inc
Reading profile /etc/firejail/disable-exec.inc
Reading profile /etc/firejail/disable-interpreters.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc
Reading profile /etc/firejail/disable-xdg.inc
Reading profile /etc/firejail/whitelist-usr-share-common.inc
Reading profile /etc/firejail/whitelist-var-common.inc
Parent pid 1140388, child pid 1140389
47 programs installed in 100.35 ms
Warning fcopy: skipping /etc/alternatives/fakeroot, cannot find inode
Warning fcopy: skipping /etc/alternatives/cssparse, cannot find inode
Warning fcopy: skipping /etc/alternatives/vim, cannot find inode
Warning fcopy: skipping /etc/alternatives/display-im6, cannot find inode
Warning fcopy: skipping /etc/alternatives/wine.fr.1.gz, cannot find inode
Warning fcopy: skipping /etc/alternatives/wineconsole, cannot find inode
Warning fcopy: skipping /etc/alternatives/pinentry-x11, cannot find inode
Warning fcopy: skipping /etc/alternatives/proxychains, cannot find inode
Warning fcopy: skipping /etc/alternatives/winepath, cannot find inode
Warning fcopy: skipping /etc/alternatives/sar, cannot find inode
Error fcopy: size limit of 500 MB reached
Warning: skipping apache for private /etc
Warning: skipping crypto-policies for private /etc
Warning: skipping youtube-dl.conf for private /etc
Private /etc installed in 43.88 ms
Blacklist violations are logged to syslog
Child process initialized in 270.84 ms
[youtube] 7RMz7tePA98: Downloading webpage
[youtube] 7RMz7tePA98: Downloading MPD manifest
[dashsegments] Total fragments: 15
[download] Destination: Firejail Introduction-7RMz7tePA98.f136.mp4
[download] 100% of 4.36MiB in 00:03
[dashsegments] Total fragments: 9
[download] Destination: Firejail Introduction-7RMz7tePA98.f140.m4a
[download] 100% of 1.14MiB in 00:01
[ffmpeg] Merging formats into "Firejail Introduction-7RMz7tePA98.mp4"
ERROR: ffmpeg: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory

Parent is shutting down, bye...
rusty-snake commented 3 years ago

I'm closing here due to inactivity, please fell free to request to reopen if you still have this issue.

hyiltiz commented 3 years ago

I am willing to offer any help in debugging as I've been doing, but cannot resolve the issue alone by myself and the issue still persists.

rusty-snake commented 3 years ago

I can't follow this thread. Can you give a short summary.