Closed murmur9283 closed 1 year ago
AmericanPresidentJimmyCarter
commented
1 year ago Looking at the trace, you are not on latest because a couple of the recent hotfixes are not patched. Please ensure you're up to date on dalle-flow at upstream:
https://github.com/jina-ai/dalle-flow
The present HEAD for main is e3da4fd41207617837b85ecbf908a888319363f6. The image is too big for dockerhub so you will need to build it on its own.
murmur9283
commented
1 year ago Thank you very much for your help. I cleaned up my installation, deleted the previous containers and installed it as you recommended.
Building went without errors. But when I try to run it I ge docker run -e ENABLE_STABLE_DIFFUSION="1" -e DISABLE_DALLE_MEGA="1" -e DISABLE_GLID3XL="1" -p 51005:51005 -it -v /home/pat/ldm:/dalle/stable-diffusion/models/ldm/ -v $HOME/.cache:/home/dalle/.cache --gpus all jinaai/dalle-flow Latent diffusion checkpoints will not be downloaded because DISABLE_GLID3XL flag is on
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
/dalle/dalle-flow/env/bin/jina flow --uses flow.tmp.yml ╭──────────────┬──────────────────────────────────╮ │ Argument │ Value │ ├──────────────┼──────────────────────────────────┤ │ cli │ flow │ │ env │ None │ │ inspect │ COLLECT │ │ log-config │ default │ │ name │ None │ │ quiet │ False │ │ quiet-error │ False │ │ uses │ flow.tmp.yml │ │ workspace │ None │ │ workspace-id │ 47955b84249046fe91f0dbf86e0437e9 │ ╰──────────────┴──────────────────────────────────╯ Docker daemon doesn't seem to be running. Please run the Docker daemon and try again.
Which is wicked as it seems to be an output from within docker. I am running ubuntu on WSL and docker desktop - which has a couple of containers running. Also docker ps returns them in the shell - I can rule out that the service is down. Sorry for the trouble - and thank you again
AmericanPresidentJimmyCarter
commented
1 year ago It sounds like the image is fine, but I am unsure about the "Docker daemon doesn't seem to be running. Please run the Docker daemon and try again." It reports no other errors and then closes? Or does the container stay open after?
murmur9283
commented
1 year ago No - no other errors and closes right after the error. first time I started it abter building the image it downloaded some resources but did shut down right after.
AmericanPresidentJimmyCarter
commented
1 year ago To confirm, you're running nvidia-docker over WSL2?
murmur9283
commented
1 year ago Yes to both:
WSL C:\Users\patri> wsl -l -v NAME STATE VERSION
Nvidia: sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi Fri Nov 18 17:11:05 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.46 Driver Version: 526.86 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:09:00.0 Off | N/A | | 0% 29C P8 10W / 280W | 0MiB / 11264MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA GeForce ... On | 00000000:0D:00.0 On | N/A | | 0% 35C P8 25W / 350W | 1575MiB / 24576MiB | 5% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
AmericanPresidentJimmyCarter
commented
1 year ago I wonder if this is the issue? Try the fix from there and see if that fixes it. https://stackoverflow.com/questions/70409182/python-docker-client-is-not-able-to-find-docker-daemon
murmur9283
commented
1 year ago Hum.. good idea. But I already was in the docker group. docker:x:120:root,pat thing is also I did not get that error with the docker image I pulled from dockerhub. weird
AmericanPresidentJimmyCarter
commented
1 year ago It sounds like the issue is inside the container -- did you check there? You can shell into the container by adding -it --entrypoint /bin/bash before the image you wish to run. I think jina might use docker itself to run the executors within the container.
AmericanPresidentJimmyCarter
commented
1 year ago Another thing to ensure is that you're building it with the user permissions and not docker build .. The image should be built with docker build --build-arg GROUP_ID=$(id -g ${USER}) --build-arg USER_ID=$(id -u ${USER}) -t jinaai/dalle-flow ., which passes the right user permissions into it.
murmur9283
commented
1 year ago Okay.. re-built it with docker build --build-arg GROUP_ID=$(id -g ${USER}) --build-arg USER_ID=$(id -u ${USER}) -t jinaai/dalle-flow . got a shell into the container and ran ./start.sh from there. the initial startup is currently re-fetching mega-1 and the other artifacts but I already can see on the second line of the log that the problem persists. ╭──────────────┬──────────────────────────────────╮ │ Argument │ Value │ ├──────────────┼──────────────────────────────────┤ │ cli │ flow │ │ env │ None │ │ inspect │ COLLECT │ │ log-config │ default │ │ name │ None │ │ quiet │ False │ │ quiet-error │ False │ │ uses │ flow.tmp.yml │ │ workspace │ None │ │ workspace-id │ 70273897a2174e158ed0bef6e3a31556 │ ╰──────────────┴──────────────────────────────────╯ ⠋ Fetching CLIPTorchEncoder from Jina Hub ...🔐 You are not logged in to Jina AI. To log in, use jina auth login or set env variable JINA_AUTH_TOKEN. Docker daemon doesn't seem to be running. Please run the Docker daemon and try again.
AmericanPresidentJimmyCarter
commented
1 year ago Yeah the environment variables must not be set. You can alter what you wish to run/not run in /dalle/dalle-flow/flow.tmp.yml then run jina flow --uses flow.tmp.yml.
Inside the container itself, is the default user dalle in the docker group?
murmur9283
commented
1 year ago Hum.. so from within the container I cannot find any trace of docker. /etc/group users:x:100: nogroup:x:65534: systemd-timesync:x:101: systemd-journal:x:102: systemd-network:x:103: systemd-resolve:x:104: messagebus:x:105: dalle:x:1000:
and also looking for files matching docker I cannot find anything that would indicate that docker was present.
AmericanPresidentJimmyCarter
commented
1 year ago And I presume leaving the shell and attempting to bring up the docker image for your new build also does not work, with the same error?
At this point I might say that your best bet is either opening an issue in the dalle-flow upstream or contacting jina on their slack server (https://jina-ai.slack.com). Unfortunately I am unsure why it is not working.
murmur9283
commented
1 year ago yeah.. even tried to launch it via docker-desktop. Same errors in the log. I will dig around a bit longer and see if I find a way to bind to the docker.sock on the host and if I fail I will so as you suggested and have a chat with the guys from Jina. Thank you very much for your help. Appeciated.
AmericanPresidentJimmyCarter
commented
1 year ago No problem! I am sorry we have not been able to get it working. Worst case you can try a native installation and see if that works for you.
PascalTemel
commented
1 year ago Just chiming in, I had the same issue and was able to fix it by building the container manually with the latest commits
murmur9283
commented
1 year ago Awesone.. Thank you for posting here. I will try again
murmur9283
commented
1 year ago Works like a charm. Thank you for posting the update - I would not have tried so soon. I guess this issue can be closed.
AmericanPresidentJimmyCarter
commented
1 year ago I will remove references to the docker image and tell users to build the image themselves. Thanks!
It looks as if various prerequisites are not satisfied in the latest build. I deleted the old docker I was running a couple of months back and deleted the .cache directory to make sure I didn't have any legacies. Got the same issues when building the container myself.
2022-11-17 18:05:11 Latent diffusion checkpoints will not be downloaded because DISABLE_GLID3XL flag is on 2022-11-17 18:05:11 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMNNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNNNNNNMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMWNNNWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMWxxxxxxxxxOMMMMMNxxxxxxxxx0MMMMMKddddddxkKWMMMMMMMMMMMMXOxdddxONMMMM 2022-11-17 18:05:11 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllo0MMMMMMMM0olllllllllo0MM 2022-11-17 18:05:11 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOlllllllllllloWMMMMMdllllllllllllldM 2022-11-17 18:05:11 MMMMMMMMMMMMXllllllllldMMMMM0lllllllllxMMMMMOllllllllllllloMMMM0lllllllllllllllK 2022-11-17 18:05:11 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllllKMMM0lllllllllllllllO 2022-11-17 18:05:11 MMMMMMMMMMMMKllllllllldMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMollllllllllllllO 2022-11-17 18:05:11 MWOkkkkk0MMMKlllllllllkMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMxlllllllllllllO 2022-11-17 18:05:11 NkkkkkkkkkMMKlllllllloMMMMMM0lllllllllxMMMMMOllllllllllllll0MMMMMMWOdolllllllllO 2022-11-17 18:05:11 KkkkkkkkkkNMKllllllldMMMMMMMMWWWWWWWWWMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MOkkkkkkk0MMKllllldXMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 MMWX00KXMMMMXxk0XMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 2022-11-17 18:05:11 2022-11-17 18:05:11 2022-11-17 18:05:11 /dalle/dalle-flow/env/bin/jina flow --uses", line 883, in exec_module", line 241, in _call_with_frames_removed", line 883, in exec_module", line 241, in _call_with_frames_removed
2022-11-17 18:05:50 sys.exit(main())
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jinacli/init.py", line 143, in main
2022-11-17 18:05:50 getattr(api, args.cli.replace('-', ''))(args)
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina_cli/api.py", line 169, in flow
2022-11-17 18:05:50 with f:
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/flow/base.py", line 1556, in enter
2022-11-17 18:05:50 return self.start()
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/flow/builder.py", line 33, in arg_wrapper
2022-11-17 18:05:50 return func(self, *args, kwargs)
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/flow/base.py", line 1620, in start
2022-11-17 18:05:50 self._wait_until_all_ready()
2022-11-17 18:05:50 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/flow/base.py", line 1719, in _wait_until_all_ready
2022-11-17 18:05:50 raise RuntimeFailToStart
2022-11-17 18:05:50 jina.excepts.RuntimeFailToStart
2022-11-17 18:05:11 flow.tmp.yml
2022-11-17 18:05:11 ╭──────────────┬──────────────────────────────────╮ 2022-11-17 18:05:11 │ Argument │ Value │ 2022-11-17 18:05:11 ├──────────────┼──────────────────────────────────┤ 2022-11-17 18:05:11 │ cli │ flow │ 2022-11-17 18:05:11 │ env │ None │ 2022-11-17 18:05:11 │ inspect │ COLLECT │ 2022-11-17 18:05:11 │ log-config │ default │ 2022-11-17 18:05:11 │ name │ None │ 2022-11-17 18:05:11 │ quiet │ False │ 2022-11-17 18:05:11 │ quiet-error │ False │ 2022-11-17 18:05:11 │ uses │ flow.tmp.yml │ 2022-11-17 18:05:11 │ workspace │ None │ 2022-11-17 18:05:11 │ workspace-id │ 0d4291383be94155a9804ff73b05d755 │ 2022-11-17 18:05:11 ╰──────────────┴──────────────────────────────────╯ ⠋ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/0 -:--:--DEBUG gateway/rep-0@35 adding connection for deployment stable/heads/0 to grpc://0.0.0.0:56506 [11/17/22 17:05:13] 2022-11-17 18:05:13 DEBUG store/rep-0@32 <executor.DalleFlowStore object at 0x7ffa73777b20> is successfully loaded! [11/17/22 17:05:13] 2022-11-17 18:05:13 DEBUG gateway/rep-0@35 adding connection for deployment clipseg/heads/0 to grpc://0.0.0.0:55596
2022-11-17 18:05:13 DEBUG gateway/rep-0@35 adding connection for deployment upscaler/heads/0 to grpc://0.0.0.0:55658
2022-11-17 18:05:13 DEBUG gateway/rep-0@35 adding connection for deployment store/heads/0 to grpc://0.0.0.0:54488
2022-11-17 18:05:13 DEBUG store/rep-0@32 start listening on 0.0.0.0:54488
2022-11-17 18:05:13 DEBUG gateway/rep-0@35 start server bound to 0.0.0.0:51005
⠙ Waiting stable clipseg upscaler store gateway summary... ━━━━━━━━━━━━━━━━━━━━━ DEBUG store/rep-0@10 ready and listening [11/17/22 17:05:13] 2022-11-17 18:05:13 ⠙ Waiting stable clipseg upscaler store gateway summary... ━ DEBUG gateway/rep-0@10 ready and listening [11/17/22 17:05:13] 2022-11-17 18:05:13 ⠙ Waiting stable clipseg upscaler store gateway summary... ━ ⠹ Waiting stable clipseg upscaler store gateway summary... ━━━━━━━━━━━━━━━━━━━━━ ⠸ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ ⠴ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ ⠦ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ ⠧ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00:00DeprecationWarning: invalid escape sequence '\:' (raised from /dalle/stable-diffusion/src/stable_inference/util.py:411) 2022-11-17 18:05:13 CRITI… clipseg/rep-0@28 can not load the executor from executors/clipseg/config.yml [11/17/22 17:05:13] 2022-11-17 18:05:13 ERROR clipseg/rep-0@28 ImportError('can not import module from /dalle/dalle-flow/executors/clipseg/executor.py') during <class 'jina.serve.runtimes.worker.WorkerRuntime'> initialization [11/17/22 17:05:13] 2022-11-17 18:05:13 add "--quiet-error" to suppress the exception details
2022-11-17 18:05:13 Traceback (most recent call last):
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 128, in _path_import
2022-11-17 18:05:13 spec.loader.exec_module(module)
2022-11-17 18:05:13 File "
2022-11-17 18:05:13 File "
2022-11-17 18:05:13 File "/dalle/dalle-flow/executors/clipseg/executor.py", line 17, in
2022-11-17 18:05:13 from clipseg.clipseg import CLIPDensePredT
2022-11-17 18:05:13 ModuleNotFoundError: No module named 'clipseg'
2022-11-17 18:05:13
2022-11-17 18:05:13 The above exception was the direct cause of the following exception:
2022-11-17 18:05:13
2022-11-17 18:05:13 Traceback (most recent call last):
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/pods/init.py", line 74, in run
2022-11-17 18:05:13 runtime = runtime_cls(
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/worker/init.py", line 36, in init
2022-11-17 18:05:13 super().init(args, *kwargs)
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/asyncio.py", line 80, in init
2022-11-17 18:05:13 self._loop.run_until_complete(self.async_setup())
2022-11-17 18:05:13 File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
2022-11-17 18:05:13 return future.result()
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/worker/init.py", line 101, in async_setup
2022-11-17 18:05:13 self._data_request_handler = DataRequestHandler(
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/request_handlers/data_request_handler.py", line 49, in init
2022-11-17 18:05:13 self._load_executor(
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/request_handlers/data_request_handler.py", line 139, in _load_executor
2022-11-17 18:05:13 self._executor: BaseExecutor = BaseExecutor.load_config(
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/jaml/init.py", line 730, in load_config
2022-11-17 18:05:13 load_py_modules(
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/jaml/helper.py", line 285, in load_py_modules
2022-11-17 18:05:13 PathImporter.add_modules(mod)
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 159, in add_modules
2022-11-17 18:05:13 _path_import(complete_path(m))
2022-11-17 18:05:13 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 130, in _path_import
2022-11-17 18:05:13 raise ImportError(f'can not import module from {absolute_path}') from ex
2022-11-17 18:05:13 ImportError: can not import module from /dalle/dalle-flow/executors/clipseg/executor.py
2022-11-17 18:05:13 DEBUG clipseg/rep-0@28 process terminated
⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ DEBUG clipseg/rep-0@10 waiting for ready or shutdown signal from runtime [11/17/22 17:05:13] 2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 shutdown is is already set. Runtime will end gracefully on its own
2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 terminating the runtime process
2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 runtime process properly terminated
2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 terminated
2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 joining the process
2022-11-17 18:05:13 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ DEBUG clipseg/rep-0@10 successfully joined the process
2022-11-17 18:05:14 ⠇ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━ ⠋ Waiting stable clipseg upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ ⠙ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/ ⠹ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/ ⠸ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00:01CRITI… stable/rep-0@27 can not load the executor from executors/stable/config.yml [11/17/22 17:05:14] 2022-11-17 18:05:14 ERROR stable/rep-0@27 ImportError('can not import module from /dalle/dalle-flow/executors/stable/executor.py') during <class 'jina.serve.runtimes.worker.WorkerRuntime'> initialization [11/17/22 17:05:14] 2022-11-17 18:05:14 add "--quiet-error" to suppress the exception details
2022-11-17 18:05:14 Traceback (most recent call last):
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 128, in _path_import
2022-11-17 18:05:14 spec.loader.exec_module(module)
2022-11-17 18:05:14 File "
2022-11-17 18:05:14 File "
2022-11-17 18:05:14 File "/dalle/dalle-flow/executors/stable/executor.py", line 14, in
2022-11-17 18:05:14 from stable_inference import StableDiffusionInference
2022-11-17 18:05:14 File "/dalle/stable-diffusion/src/stable_inference/init.py", line 1, in
2022-11-17 18:05:14 from . import exceptions, util
2022-11-17 18:05:14 File "/dalle/stable-diffusion/src/stable_inference/util.py", line 22, in
2022-11-17 18:05:14 from ldm.modules.encoders.modules import FrozenCLIPEmbedder
2022-11-17 18:05:14 File "/dalle/stable-diffusion/src/ldm/modules/encoders/modules.py", line 6, in
2022-11-17 18:05:14 from transformers import CLIPTokenizer, CLIPTextModel
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/transformers/init.py", line 30, in
2022-11-17 18:05:14 from . import dependency_versions_check
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 17, in
2022-11-17 18:05:14 from .utils.versions import require_version, require_version_core
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/transformers/utils/init.py", line 34, in
2022-11-17 18:05:14 from .generic import (
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/transformers/utils/generic.py", line 36, in
2022-11-17 18:05:14 import jax.numpy as jnp
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jax/init.py", line 35, in
2022-11-17 18:05:14 from jax import config as _config_module
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jax/config.py", line 17, in
2022-11-17 18:05:14 from jax._src.config import config
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jax/_src/config.py", line 29, in
2022-11-17 18:05:14 from jax._src import lib
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jax/_src/lib/init.py", line 87, in
2022-11-17 18:05:14 version = check_jaxlib_version(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jax/_src/lib/init.py", line 76, in check_jaxlib_version
2022-11-17 18:05:14 raise RuntimeError(msg)
2022-11-17 18:05:14 RuntimeError: jaxlib is version 0.3.10, but this version of jax requires version >= 0.3.15.
2022-11-17 18:05:14
2022-11-17 18:05:14 The above exception was the direct cause of the following exception:
2022-11-17 18:05:14
2022-11-17 18:05:14 Traceback (most recent call last):
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/orchestrate/pods/init.py", line 74, in run
2022-11-17 18:05:14 runtime = runtime_cls(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/worker/init.py", line 36, in init
2022-11-17 18:05:14 super().init(args, kwargs)
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/asyncio.py", line 80, in init
2022-11-17 18:05:14 self._loop.run_until_complete(self.async_setup())
2022-11-17 18:05:14 File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
2022-11-17 18:05:14 return future.result()
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/worker/init.py", line 101, in async_setup
2022-11-17 18:05:14 self._data_request_handler = DataRequestHandler(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/request_handlers/data_request_handler.py", line 49, in init
2022-11-17 18:05:14 self._load_executor(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/serve/runtimes/request_handlers/data_request_handler.py", line 139, in _load_executor
2022-11-17 18:05:14 self._executor: BaseExecutor = BaseExecutor.load_config(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/jaml/init.py", line 730, in load_config
2022-11-17 18:05:14 load_py_modules(
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/jaml/helper.py", line 285, in load_py_modules
2022-11-17 18:05:14 PathImporter.add_modules(*mod)
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 159, in add_modules
2022-11-17 18:05:14 _path_import(complete_path(m))
2022-11-17 18:05:14 File "/dalle/dalle-flow/env/lib/python3.10/site-packages/jina/importer.py", line 130, in _path_import
2022-11-17 18:05:14 raise ImportError(f'can not import module from {absolute_path}') from ex
2022-11-17 18:05:14 ImportError: can not import module from /dalle/dalle-flow/executors/stable/executor.py
2022-11-17 18:05:14 DEBUG stable/rep-0@27 process terminated
⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/ DEBUG stable/rep-0@10 waiting for ready or shutdown signal from runtime [11/17/22 17:05:14] 2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 shutdown is is already set. Runtime will end gracefully on its own
2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 terminating the runtime process
2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 runtime process properly terminated
2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 terminated
2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 joining the process
2022-11-17 18:05:14 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ DEBUG stable/rep-0@10 successfully joined the process
2022-11-17 18:05:21 ⠴ Waiting stable upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━ ⠦ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠧ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠏ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠋ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠙ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠹ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠼ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠴ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠦ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠧ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠏ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠋ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠙ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠹ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠼ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠴ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠦ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠇ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠏ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠋ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠙ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠸ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠼ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠴ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠦ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠇ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠏ Waiting upscaler summary... ━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6 0:00: ⠦ Waiting upscaler... ━━━━━━━━━━━━━━━━━2022-11-17T17:05:21.538749412Z ━━━╺━━━━━━ ⠏ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 3/6 2022-11-17T17 ⠸ Waiting upscaler... ━━━━━━━━━━━━��2022-11-17T17:05:21.538749412Z �━━━━━━━╺━━━━ ⠴ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 2022-11-17T17:05: ⠋ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 3/6 0:00:35downloading model ../SwinIR/model_zoo/swinir/003_realSR_BSRGAN_DFOWMFC_s64w8_SwinIR-L_x4_GAN.pth ⠙ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 3/6 0:00:35UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) (raised from /dalle/dalle-flow/env/lib/python3.10/site-packages/torch/functional.py:504) ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 3/6 0:00:37DEBUG upscaler/rep-0@29 <executor.SwinIRUpscaler object at 0x7ff9ebfe2f20> is successfully loaded! [11/17/22 17:05:50] 2022-11-17 18:05:50 DEBUG upscaler/rep-0@29 start listening on 0.0.0.0:55658
DEBUG upscaler/rep-0@10 ready and listening [11/17/22 17:05:50] 2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ ERROR Flow@10 Flow is aborted due to ['stable', 'clipseg'] can not be started. [11/17/22 17:05:50] 2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ DEBUG gateway/rep-0@10 waiting for ready or shutdown signal from runtime [11/17/22 17:05:50] 2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ DEBUG gateway/rep-0@10 terminate
2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ DEBUG gateway/rep-0@10 terminating the runtime process
2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ DEBUG gateway/rep-0@10 runtime process properly terminated
2022-11-17 18:05:50 ⠼ Waiting upscaler... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0:00:37DEBUG gateway/rep-0@35 process terminated [11/17/22 17:05:50] DEBUG gateway/rep-0@10 terminated
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG gateway/rep-0@10 joining the process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG gateway/rep-0@10 successfully joined the process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG store/rep-0@10 waiting for ready or shutdown signal from runtime [11/17/22 17:05:50] 2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG store/rep-0@10 terminate
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG store/rep-0@10 terminating the runtime process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0:00:37DEBUG store/rep-0@32 cancel WorkerRuntime [11/17/22 17:05:50] DEBUG store/rep-0@10 runtime process properly terminated
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0:00:37DEBUG store/rep-0@32 stopped GRPC Server
2022-11-17 18:05:50 DEBUG store/rep-0@32 cancel WorkerRuntime
2022-11-17 18:05:50 DEBUG store/rep-0@32 stopped GRPC Server
2022-11-17 18:05:50 DEBUG store/rep-0@32 process terminated [11/17/22 17:05:50] DEBUG store/rep-0@10 terminated
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG store/rep-0@10 joining the process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG store/rep-0@10 successfully joined the process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 waiting for ready or shutdown signal from runtime
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 terminate
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 terminating the runtime process
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 runtime process properly terminated
2022-11-17 18:05:50 ⠴ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0:00:37DEBUG upscaler/rep-0@29 cancel WorkerRuntime
2022-11-17 18:05:50 DEBUG upscaler/rep-0@29 stopped GRPC Server
⠦ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0:00:37DEBUG upscaler/rep-0@29 cancel WorkerRuntime
2022-11-17 18:05:50 DEBUG upscaler/rep-0@29 stopped GRPC Server
2022-11-17 18:05:50 DEBUG upscaler/rep-0@29 process terminated [11/17/22 17:05:50] DEBUG upscaler/rep-0@10 terminated
2022-11-17 18:05:50 ⠦ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 joining the process
2022-11-17 18:05:50 ⠦ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 DEBUG upscaler/rep-0@10 successfully joined the process
2022-11-17 18:05:50 ⠦ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 4/6 0 Traceback (most recent call last): 2022-11-17 18:05:50 File "/dalle/dalle-flow/env/bin/jina", line 8, in