"Done downloading" but without any data

[tombenetin]

HI, thanks for docker container. If i run it:

:~# time docker run -e PARTS=30 -e DEBUG=1 -e ENFORCE_TOR=1 -v /fastdata/Serialy:/downloads uld-docker:latest "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso"

Debug enabled!

Starting FlareSolverr

Starting ulozto-downloader for https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso

09/24/23 17:50:46 - Done downloading

real 0m1.451s user 0m0.014s sys 0m0.023s

no more logs, no download data, container is in exited state.

what i do wrong?

thanks

[pkejval]

Hello, you're doing nothing wrong and you aren't alone with this problem. I'm investigating it. But I think it isn't container problem as it's only invoking ulozto-downloader with arguments. Seems like ulozto-downloader exits without any error message.

But yes, there is one mistake in container. Script run.sh isn't checking return code so it shows Done downloading even if ulozto-downloader download fails.

I'll investigate it further and let you know.

[pkejval]

Found out two issues when using --frontend console to solve:

1. Failed to connect to the Flaresolverr service. Make sure it's reachable or specify an alternative endpoint using the --cf-endpoint argument.

Script needs to wait for flaresolverr to start. It worked in past because we weren't caching tordb. But now start of ulozto-downloader is too fast and flaresolverr isn't started yet.

2. If file exists in /downloads folder, then ulozto-downloader fails with:

Traceback (most recent call last):
  File "/app/ulozto-downloader/ulozto-downloader.py", line 7, in <module>
    cmd.run()
  File "/app/ulozto-downloader/uldlib/cmd.py", line 157, in run
    d.download(url, args.parts, args.password, args.output, args.temp, args.yes, args.conn_timeout, args.enforce_tor)
  File "/app/ulozto-downloader/uldlib/downloader.py", line 248, in download
Cloudflare WAF detected, initializing automated Cloudflare Solver (timeout 90s).
Downloading into: '/downloads/file.rar'
    answer = self.frontend.prompt(
  File "/app/ulozto-downloader/uldlib/frontend.py", line 125, in prompt
WARNING: File '/downloads/file.rar' already exists, overwrite it? [y/n] Terminating download. Please wait for stopping all threads.
Download terminated.
    return input().strip()
EOFError: EOF when reading a line

3. When ulozto-downloader fails with error it still returns return code 0 so we can't check for it....

[tombenetin]

Hi, thanks for reply, so is there some solution how to using container? thanks

[pkejval]

Yes. Working on it and will be commiting it in few hours.

EDIT: Just pushing hotfix with sleep 10 seconds before starting ulozto-downloader. Should solve your issue and I'll try to make it more robust later.

[tombenetin]

i tried rebuild it, but same issues, but it takes +10seconds longer, but no download starting, no downloaded data:

time docker run -e PARTS=10 -e DEBUG=1 -v /fastdata/Serialy:/downloads uld-docker:latest "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso"
Debug enabled!

Starting FlareSolverr

Starting ulozto-downloader for https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso

09/25/23 06:50:22 - Done downloading 

real    0m11.475s
user    0m0.015s
sys 0m0.030s

[pkejval]

I could reproduce it myself. It happens to me only occasionally (1 of 50 tries) but error is clear: Failed to connect to the Flaresolverr service. Make sure it's reachable or specify an alternative endpoint using the --cf-endpoint argument.

This error is not visible when ulozto-downloader is started with --frontend JSON argument that's why you see only starting-done messages.

I need to figure out how to detect if Flaresolverr is started OK and listening...

[pkejval]

Added waiting for Flaresolverr start by checking its /health endpoint. I hope it will fix this issue. Please test and let me know. :)

[tombenetin]

i tested it, it looks different but with same results:

time docker run -e PARTS=10 -e DEBUG=1 -v /fastdata/Serialy:/downloads uld-docker:latest "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso"
Debug enabled!

Starting FlareSolverr and waiting to its startup

Flaresolverr still not ready! Waiting 1 second more. 60 tries remaining before exit.
2023-09-25 08:23:36 INFO     FlareSolverr 3.3.6
2023-09-25 08:23:36 INFO     Testing web browser installation...
2023-09-25 08:23:36 INFO     Platform: Linux-4.19.0-24-amd64-x86_64-with-glibc2.35
2023-09-25 08:23:36 INFO     Chrome / Chromium path: /app/flaresolverr/chrome/chrome
Flaresolverr still not ready! Waiting 1 second more. 59 tries remaining before exit.
2023-09-25 08:23:37 INFO     Chrome / Chromium major version: 115
2023-09-25 08:23:37 INFO     Launching web browser...
Flaresolverr still not ready! Waiting 1 second more. 58 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 57 tries remaining before exit.
2023-09-25 08:23:39 INFO     FlareSolverr User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36
2023-09-25 08:23:39 INFO     Test successful!
2023-09-25 08:23:39 INFO     Serving on http://0.0.0.0:8191
Starting ulozto-downloader for https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso

09/25/23 08:23:40 - Done downloading 

real    0m5.970s
user    0m0.024s
sys 0m0.021s

[pkejval]

I really don't know now. Everything is working on my Windows and Linux machines.

1. Was this container working for you before?
2. There were some fixes in ulozto-downloader branch. Could you try rebuild image with --no-cache arg or try image from dockerhub pkejval/uld-docker:main?

[tombenetin]

i tried to rebuild with --no-cache, i edit just run.sh, in others files there is no more edits:

/home/uld-docker-main# docker build --tag uld-docker --no-cache .
[+] Building 231.8s (14/14) FINISHED                                                                                                                                                                    
 => [internal] load .dockerignore                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                               0.0s
 => => transferring dockerfile: 1.13kB                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                   1.0s
 => [internal] load build context                                                                                                                                                                  0.0s
 => => transferring context: 28B                                                                                                                                                                   0.0s
 => CACHED [1/9] FROM docker.io/library/ubuntu:latest@sha256:aabed3296a3d45cede1dc866a24476c4d7e093aa806263c27ddaadbdce3c1054                                                                      0.0s
 => [2/9] RUN mkdir /app && mkdir /downloads                                                                                                                                                       0.6s
 => [3/9] WORKDIR /app                                                                                                                                                                             0.0s
 => [4/9] RUN apt update &&  apt install -y git tor python3 python3-pip jq wget gzip &&  apt install -y chromium-browser xvfb libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libxkbcommon-x11-0  78.0s
 => [5/9] RUN wget https://github.com/FlareSolverr/FlareSolverr/releases/latest/download/flaresolverr_linux_x64.tar.gz &&  tar xvzf flaresolverr_linux_x64.tar.gz &&  rm flaresolverr_linux_x64.  15.5s
 => [6/9] RUN git clone https://github.com/filo891/ulozto-downloader.git &&  cd /app/ulozto-downloader &&  git checkout remotes/origin/cfsolver-support &&  sed -i 's/\[auto-captcha\]//g' requi  90.6s 
 => [7/9] RUN wget https://raw.githubusercontent.com/torproject/tor/main/src/config/geoip -O /app/geoip.db &&  wget https://raw.githubusercontent.com/torproject/tor/main/src/config/geoip6 -O /a  2.0s 
 => [8/9] COPY ./run.sh /app                                                                                                                                                                       0.2s 
 => [9/9] RUN chmod 755 /app/run.sh                                                                                                                                                                0.4s 
 => exporting to image                                                                                                                                                                            43.2s 
 => => exporting layers                                                                                                                                                                           43.1s 
 => => writing image sha256:4676676af0c46b2b6a735788911391a50cf270f5001b008d85f1f586e3fc74a2                                                                                                       0.0s 
 => => naming to docker.io/library/uld-docker                                                                                                                                                      0.0s 
root@homeserver:/home/uld-docker-main# time docker run -e PARTS=10 -e DEBUG=1 -v /fastdata/Serialy:/downloads uld-docker:latest "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso"
Debug enabled!

Starting FlareSolverr and waiting to its startup

Flaresolverr still not ready! Waiting 1 second more. 60 tries remaining before exit.
2023-09-25 08:54:08 INFO     FlareSolverr 3.3.6
2023-09-25 08:54:08 INFO     Testing web browser installation...
2023-09-25 08:54:08 INFO     Platform: Linux-4.19.0-24-amd64-x86_64-with-glibc2.35
2023-09-25 08:54:08 INFO     Chrome / Chromium path: /app/flaresolverr/chrome/chrome
Flaresolverr still not ready! Waiting 1 second more. 59 tries remaining before exit.
2023-09-25 08:54:08 INFO     Chrome / Chromium major version: 115
2023-09-25 08:54:08 INFO     Launching web browser...
Flaresolverr still not ready! Waiting 1 second more. 58 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 57 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 56 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 55 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 54 tries remaining before exit.
Flaresolverr still not ready! Waiting 1 second more. 53 tries remaining before exit.
2023-09-25 08:54:15 INFO     FlareSolverr User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36
2023-09-25 08:54:15 INFO     Test successful!
2023-09-25 08:54:15 INFO     Serving on http://0.0.0.0:8191
Starting ulozto-downloader for https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso

09/25/23 08:54:17 - Done downloading 

real    0m11.323s
user    0m0.028s
sys 0m0.021s

[pkejval]

Looks like that problem is with ulozto-downloader itself.

Could you run container with docker run --entrypoint /bin/bash -it -e PARTS=10 -e DEBUG=1 -v /fastdata/Serialy:/downloads uld-docker:latest and issue these commands and send output here?

./ulozto-downloader/ulozto-downloader.py --auto-captcha 'https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso'

lscpu

[tombenetin]

sure:

/home/uld-docker-main# docker run --entrypoint /bin/bash -it -e PARTS=10 -e DEBUG=1 -v /fastdata/Serialy:/downloads uld-docker:latest
root@28a741428718:/app# ./ulozto-downloader/ulozto-downloader.py --auto-captcha 'https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso'
ňIllegal instruction (core dumped)
root@28a741428718:/app# ./ulozto-downloader/ulozto-downloader.py --auto-captcha 'https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso'
Illegal instruction (core dumped)
root@28a741428718:/app# ./ulozto-downloader/ulozto-downloader.py --auto-captcha 'https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso'

lscpu
Illegal instruction (core dumped)
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         40 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               GenuineIntel
  Model name:            Common KVM processor
    CPU family:          15
    Model:               6
    Thread(s) per core:  1
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            4799.99
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2
                         apic hypervisor lahf_lm cpuid_fault pti
Virtualization features: 
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    32 MiB (8 instances)
  L3:                    16 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:         
  Itlb multihit:         KVM: Vulnerable
  L1tf:                  Mitigation; PTE Inversion
  Mds:                   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Unknown: No mitigations
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[pkejval]

Yeah, that's the problem. Your CPU doesn't support AVX instructions needed for tensorflow, see https://github.com/pkejval/uld-docker/issues/3

Because you have KVM you could try to emulate AVX with qemu. But I don't have any experience with that so you are on your own there.

Another option is to compile tensorflow without AVX when building image. I tried in #no-avx branch but had no luck and interest in it. I'll welcome PR if you try to solve it.

[tombenetin]

looks that you're right: i changed my CPU architecture and now it's works. if i run docker. Thanks so much for your patience

root@b46e085a25fe:/app# ./ulozto-downloader/ulozto-downloader.py --auto-captcha 'https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso'
2023-09-25 10:15:41.970603: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-09-25 10:15:42.037362: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-09-25 10:15:43.408908: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Failed to connect to the Flaresolverr service. Make sure it's reachable or specify an alternative endpoint using the --cf-endpoint argument.

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
    CPU family:          6
    Model:               62
    Thread(s) per core:  1
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            4
    BogoMIPS:            4799.99
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xt
                         opology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti
                          ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms xsaveopt arat umip md_clear arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    32 MiB (8 instances)
  L3:                    16 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
  Mds:                   Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Unknown: No mitigations
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[pkejval]

That looks promising. Now it works?

[tombenetin]

yes, now it works <3

2023-09-25 10:28:17 INFO     127.0.0.1 POST http://127.0.0.1:8191/v1 200 OK
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.36 MB", "percent": "99.78 %", "avg_speed": "1.42 MB/s", "curr_speed": "0.22 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.36 MB", "percent": "99.78 %", "avg_speed": "1.42 MB/s", "curr_speed": "0.16 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.53 MB", "percent": "99.84 %", "avg_speed": "1.41 MB/s", "curr_speed": "0.19 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.53 MB", "percent": "99.84 %", "avg_speed": "1.41 MB/s", "curr_speed": "0.16 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.71 MB", "percent": "99.90 %", "avg_speed": "1.41 MB/s", "curr_speed": "0.19 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.71 MB", "percent": "99.90 %", "avg_speed": "1.41 MB/s", "curr_speed": "0.18 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.88 MB", "percent": "99.96 %", "avg_speed": "1.40 MB/s", "curr_speed": "0.16 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "290.88 MB", "percent": "99.96 %", "avg_speed": "1.40 MB/s", "curr_speed": "0.16 MB/s", "remaining": "0:00:00"}
{"status": "downloading", "file": "debian-9.6.0-amd64-netinst.iso", "url": "https://ulozto.cz/file/TKvQVDFBEhtL/debian-9-6-0-amd64-netinst-iso", "size": "291.0 MB", "size_float": 305135616, "downloaded": "291.00 MB", "percent": "100.00 %", "avg_speed": "1.40 MB/s", "curr_speed": "0.18 MB/s", "remaining": "0:00:00"}
{"status": "completed", "duration": "0:03:28", "avg_speed": "1.4 MB/s"}
09/25/23 10:28:17 - Done downloading 

real    3m56.010s
user    0m0.048s
sys 0m0.083s

[pkejval]

Sweet! 👍