search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
313 stars 102 forks source link

AWS related question #1169

Closed sergii-mamedov closed 8 months ago

sergii-mamedov commented 9 months ago

I'm migrating a pipeline on AWS Cloud. When testing one of the steps of the pipeline that is launched in AWS Lambda, I encountered the following exception.

    Traceback (most recent call last):
      File "/function/lithops/worker/jobrunner.py", line 236, in run
        result = func(**data)
      File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 61, in wrapper_func
      File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/build_moldb.py", line 135, in _build_moldb
      File "/tmp/lithops-root/modules/642ab9-0-M000/sm/engine/annotation_lithops/build_moldb.py", line 77, in get_formulas_df
        with ProcessPoolExecutor() as executor:
      File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 555, in __init__
        self._call_queue = _SafeQueue(
      File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 165, in __init__
        super().__init__(max_size, ctx=ctx)
      File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 42, in __init__
        self._rlock = ctx.Lock()
      File "/usr/local/lib/python3.8/multiprocessing/context.py", line 68, in Lock
        return Lock(ctx=self.get_context())
      File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
        SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
      File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
        sl = self._semlock = _multiprocessing.SemLock(
    FileNotFoundError: [Errno 2] No such file or directory

This is about the _build_moldb function, and specifically about this block of code. As far as I understand, previously AWS Lambda did not support shared memory for processes. I have not been able to find any more recent information on this. @JosepSampe may you have encountered this before.

JosepSampe commented 9 months ago

Hi @sergii-mamedov! What I know regarding this is the same as you found. Maybe @aitorarjona knows more about this

aitorarjona commented 9 months ago

Hi @sergii-mamedov , as far as I know, Lambda still does not support shared memory (/dev/shm). Some Python multiprocessing code does not work. You could try replacing the Process pool by spawning individual processes and sharing data trough multiprocessing Pipes.

sergii-mamedov commented 9 months ago
  1. I need to run AWS EC2 with different amount of RAM for one of the pipeline steps. We are currently using consume mode. As far as I understand, there is no good approach to combine my requirement with the consume mode approach? And the correct option is to use start/reuse mode with the ability to select master_instance_type?
  2. Is it possible for AWS EC2 to use Docker image approach like it works in IBM Cloud?
JosepSampe commented 9 months ago

Regarding 1: Need to check what is the best option for you. Do you prefer having 2 different VMs and work with both in the consume mode? or do you prefer the VMs to be automatically created/deleted every time?

Regarding 2: You can try this option. You can create a script myscript.sh and include the command docker pull registry/imagename:version and then run command: lithops image build -b aws_ec2 -f myscript.sh custom-lithops-vm-image. This will include your docker image in your VM image as in the IBM Cloud example. In myscript.sh you can include any ubuntu command you want to run in order to customize your VM image.

sergii-mamedov commented 9 months ago

I have a problem with the initial installation of lithops on EC2. As far as I can see, this function is responsible for this. The complete log located in the file /tmp/lithops-root/service.log is given below:

debconf: falling back to frontend: Readline
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
19 added, 6 removed; done.
Setting up python3-software-properties (0.99.9.12) ...
Setting up gnupg-agent (2.2.19-3ubuntu2.2) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.20) ...
Setting up curl (7.68.0-1ubuntu2.20) ...
Setting up software-properties-common (0.99.9.12) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.3) ...
Processing triggers for ca-certificates (20230311ubuntu0.20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:6 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [33.3 kB]
Fetched 90.9 kB in 0s (323 kB/s)
Reading package lists...
--> Checking internet connection
--> Installing Lithops system dependencies
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://download.docker.com/linux/ubuntu focal InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-9
  docker-buildx-plugin docker-ce-rootless-extras docker-compose-plugin
  dpkg-dev fakeroot g++ g++-9 gcc gcc-10-base gcc-9 gcc-9-base
  libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
  libasan5 libatomic1 libbinutils libc-dev-bin libc6 libc6-dev libcc1-0
  libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl libexpat1-dev libfakeroot
  libfile-fcntllock-perl libgcc-9-dev libgcc-s1 libgomp1 libisl22 libitm1
  liblsan0 libmpc3 libpython3-dev libpython3.8 libpython3.8-dev
  libpython3.8-minimal libpython3.8-stdlib libquadmath0 libstdc++-9-dev
  libstdc++6 libtsan0 libubsan1 linux-libc-dev make manpages-dev pigz
  python-pip-whl python3-dev python3-wheel python3.8 python3.8-dev
  python3.8-minimal slirp4netns zlib1g-dev
Suggested packages:
  binutils-doc cpp-doc gcc-9-locales aufs-tools cgroupfs-mount | cgroup-lite
  debian-keyring g++-multilib g++-9-multilib gcc-9-doc gcc-multilib autoconf
  automake libtool flex bison gdb gcc-doc gcc-9-multilib glibc-doc bzr
  libstdc++-9-doc make-doc python3.8-venv python3.8-doc binfmt-support zip
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential
  containerd.io cpp cpp-9 docker-buildx-plugin docker-ce docker-ce-cli
  docker-ce-rootless-extras docker-compose-plugin dpkg-dev fakeroot g++ g++-9
  gcc gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libasan5 libatomic1 libbinutils libc-dev-bin
  libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdpkg-perl
  libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev libgomp1
  libisl22 libitm1 liblsan0 libmpc3 libpython3-dev libpython3.8-dev
  libquadmath0 libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make
  manpages-dev pigz python-pip-whl python3-dev python3-pip python3-wheel
  python3.8-dev slirp4netns unzip zlib1g-dev
The following packages will be upgraded:
  gcc-10-base libc6 libgcc-s1 libpython3.8 libpython3.8-minimal
  libpython3.8-stdlib libstdc++6 python3.8 python3.8-minimal
9 upgraded, 59 newly installed, 0 to remove and 98 not upgraded.
Need to get 176 MB of archives.
After this operation, 638 MB of additional disk space will be used.
Get:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 gcc-10-base amd64 10.5.0-1ubuntu1~20.04 [20.8 kB]
Get:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgcc-s1 amd64 10.5.0-1ubuntu1~20.04 [41.8 kB]
Get:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libstdc++6 amd64 10.5.0-1ubuntu1~20.04 [501 kB]
Get:4 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libc6 amd64 2.31-0ubuntu9.12 [2722 kB]
Get:5 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.6.24-1 [28.6 MB]
Get:6 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04.8 [1625 kB]
Get:7 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04.8 [387 kB]
Get:8 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04.8 [1675 kB]
Get:9 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04.8 [1902 kB]
Get:10 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04.8 [717 kB]
Get:11 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:12 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 binutils-common amd64 2.34-6ubuntu1.6 [207 kB]
Get:13 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libbinutils amd64 2.34-6ubuntu1.6 [473 kB]
Get:14 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libctf-nobfd0 amd64 2.34-6ubuntu1.6 [47.4 kB]
Get:15 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libctf0 amd64 2.34-6ubuntu1.6 [46.6 kB]
Get:16 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 binutils-x86-64-linux-gnu amd64 2.34-6ubuntu1.6 [1613 kB]
Get:17 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 binutils amd64 2.34-6ubuntu1.6 [3376 B]
Get:18 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libc-dev-bin amd64 2.31-0ubuntu9.12 [71.6 kB]
Get:19 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-165.182 [1112 kB]
Get:20 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libcrypt-dev amd64 1:4.4.10-10ubuntu4 [104 kB]
Get:21 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libc6-dev amd64 2.31-0ubuntu9.12 [2519 kB]
Get:22 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 gcc-9-base amd64 9.4.0-1ubuntu1~20.04.2 [18.9 kB]
Get:23 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libisl22 amd64 0.22.1-1 [592 kB]
Get:24 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libmpc3 amd64 1.1.0-1 [40.8 kB]
Get:25 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 cpp-9 amd64 9.4.0-1ubuntu1~20.04.2 [7502 kB]
Get:26 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 cpp amd64 4:9.3.0-1ubuntu2 [27.6 kB]
Get:27 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libcc1-0 amd64 10.5.0-1ubuntu1~20.04 [48.8 kB]
Get:28 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgomp1 amd64 10.5.0-1ubuntu1~20.04 [102 kB]
Get:29 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libitm1 amd64 10.5.0-1ubuntu1~20.04 [26.2 kB]
Get:30 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libatomic1 amd64 10.5.0-1ubuntu1~20.04 [9284 B]
Get:31 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libasan5 amd64 9.4.0-1ubuntu1~20.04.2 [2752 kB]
Get:32 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 liblsan0 amd64 10.5.0-1ubuntu1~20.04 [835 kB]
Get:33 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libtsan0 amd64 10.5.0-1ubuntu1~20.04 [2016 kB]
Get:34 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libubsan1 amd64 10.5.0-1ubuntu1~20.04 [785 kB]
Get:35 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libquadmath0 amd64 10.5.0-1ubuntu1~20.04 [146 kB]
Get:36 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libgcc-9-dev amd64 9.4.0-1ubuntu1~20.04.2 [2359 kB]
Get:37 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 gcc-9 amd64 9.4.0-1ubuntu1~20.04.2 [8276 kB]
Get:38 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 gcc amd64 4:9.3.0-1ubuntu2 [5208 B]
Get:39 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libstdc++-9-dev amd64 9.4.0-1ubuntu1~20.04.2 [1722 kB]
Get:40 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-buildx-plugin amd64 0.11.2-1~ubuntu.20.04~focal [28.2 MB]
Get:41 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 g++-9 amd64 9.4.0-1ubuntu1~20.04.2 [8421 kB]
Get:42 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 g++ amd64 4:9.3.0-1ubuntu2 [1604 B]
Get:43 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 make amd64 4.2.1-1.2 [162 kB]
Get:44 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libdpkg-perl all 1.19.7ubuntu3.2 [231 kB]
Get:45 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 dpkg-dev all 1.19.7ubuntu3.2 [679 kB]
Get:46 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 build-essential amd64 12.8ubuntu1.1 [4664 B]
Get:47 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libfakeroot amd64 1.24-1 [25.7 kB]
Get:48 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 fakeroot amd64 1.24-1 [62.6 kB]
Get:49 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libalgorithm-diff-perl all 1.19.03-2 [46.6 kB]
Get:50 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6 [11.3 kB]
Get:51 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
Get:52 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libexpat1-dev amd64 2.2.9-1ubuntu0.6 [116 kB]
Get:53 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libfile-fcntllock-perl amd64 0.22-3build4 [33.1 kB]
Get:54 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04.8 [3950 kB]
Get:55 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libpython3-dev amd64 3.8.2-0ubuntu2 [7236 B]
Get:56 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 manpages-dev all 5.05-1 [2266 kB]
Get:57 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-cli amd64 5:24.0.6-1~ubuntu.20.04~focal [13.3 MB]
Get:58 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1805 kB]
Get:59 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-2ubuntu1.5 [155 kB]
Get:60 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04.8 [514 kB]
Get:61 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-dev amd64 3.8.2-0ubuntu2 [1212 B]
Get:62 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-wheel all 0.34.2-1ubuntu0.1 [23.9 kB]
Get:63 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]
Get:64 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 slirp4netns amd64 0.4.3-1 [74.3 kB]
Get:65 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 unzip amd64 6.0-25ubuntu1.1 [168 kB]
Get:66 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce amd64 5:24.0.6-1~ubuntu.20.04~focal [22.6 MB]
Get:67 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-rootless-extras amd64 5:24.0.6-1~ubuntu.20.04~focal [9038 kB]
Get:68 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-compose-plugin amd64 2.21.0-1~ubuntu.20.04~focal [11.9 MB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Fetched 176 MB in 5s (35.7 MB/s)
(Reading database ... 61950 files and directories currently installed.)
Preparing to unpack .../gcc-10-base_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking gcc-10-base:amd64 (10.5.0-1ubuntu1~20.04) over (10.3.0-1ubuntu1~20.04) ...
Setting up gcc-10-base:amd64 (10.5.0-1ubuntu1~20.04) ...
(Reading database ... 61950 files and directories currently installed.)
Preparing to unpack .../libgcc-s1_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libgcc-s1:amd64 (10.5.0-1ubuntu1~20.04) over (10.3.0-1ubuntu1~20.04) ...
Setting up libgcc-s1:amd64 (10.5.0-1ubuntu1~20.04) ...
(Reading database ... 61950 files and directories currently installed.)
Preparing to unpack .../libstdc++6_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libstdc++6:amd64 (10.5.0-1ubuntu1~20.04) over (10.3.0-1ubuntu1~20.04) ...
Setting up libstdc++6:amd64 (10.5.0-1ubuntu1~20.04) ...
(Reading database ... 61950 files and directories currently installed.)
Preparing to unpack .../libc6_2.31-0ubuntu9.12_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Unpacking libc6:amd64 (2.31-0ubuntu9.12) over (2.31-0ubuntu9.9) ...
Setting up libc6:amd64 (2.31-0ubuntu9.12) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
(Reading database ... 61950 files and directories currently installed.)
Preparing to unpack .../00-libpython3.8_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.8) over (3.8.10-0ubuntu1~20.04.7) ...
Preparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking python3.8 (3.8.10-0ubuntu1~20.04.8) over (3.8.10-0ubuntu1~20.04.7) ...
Preparing to unpack .../02-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.8) over (3.8.10-0ubuntu1~20.04.7) ...
Preparing to unpack .../03-python3.8-minimal_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04.8) over (3.8.10-0ubuntu1~20.04.7) ...
Preparing to unpack .../04-libpython3.8-minimal_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.8) over (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package pigz.
Preparing to unpack .../05-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package binutils-common:amd64.
Preparing to unpack .../06-binutils-common_2.34-6ubuntu1.6_amd64.deb ...
Unpacking binutils-common:amd64 (2.34-6ubuntu1.6) ...
Selecting previously unselected package libbinutils:amd64.
Preparing to unpack .../07-libbinutils_2.34-6ubuntu1.6_amd64.deb ...
Unpacking libbinutils:amd64 (2.34-6ubuntu1.6) ...
Selecting previously unselected package libctf-nobfd0:amd64.
Preparing to unpack .../08-libctf-nobfd0_2.34-6ubuntu1.6_amd64.deb ...
Unpacking libctf-nobfd0:amd64 (2.34-6ubuntu1.6) ...
Selecting previously unselected package libctf0:amd64.
Preparing to unpack .../09-libctf0_2.34-6ubuntu1.6_amd64.deb ...
Unpacking libctf0:amd64 (2.34-6ubuntu1.6) ...
Selecting previously unselected package binutils-x86-64-linux-gnu.
Preparing to unpack .../10-binutils-x86-64-linux-gnu_2.34-6ubuntu1.6_amd64.deb ...
Unpacking binutils-x86-64-linux-gnu (2.34-6ubuntu1.6) ...
Selecting previously unselected package binutils.
Preparing to unpack .../11-binutils_2.34-6ubuntu1.6_amd64.deb ...
Unpacking binutils (2.34-6ubuntu1.6) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../12-libc-dev-bin_2.31-0ubuntu9.12_amd64.deb ...
Unpacking libc-dev-bin (2.31-0ubuntu9.12) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../13-linux-libc-dev_5.4.0-165.182_amd64.deb ...
Unpacking linux-libc-dev:amd64 (5.4.0-165.182) ...
Selecting previously unselected package libcrypt-dev:amd64.
Preparing to unpack .../14-libcrypt-dev_1%3a4.4.10-10ubuntu4_amd64.deb ...
Unpacking libcrypt-dev:amd64 (1:4.4.10-10ubuntu4) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../15-libc6-dev_2.31-0ubuntu9.12_amd64.deb ...
Unpacking libc6-dev:amd64 (2.31-0ubuntu9.12) ...
Selecting previously unselected package gcc-9-base:amd64.
Preparing to unpack .../16-gcc-9-base_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking gcc-9-base:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package libisl22:amd64.
Preparing to unpack .../17-libisl22_0.22.1-1_amd64.deb ...
Unpacking libisl22:amd64 (0.22.1-1) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../18-libmpc3_1.1.0-1_amd64.deb ...
Unpacking libmpc3:amd64 (1.1.0-1) ...
Selecting previously unselected package cpp-9.
Preparing to unpack .../19-cpp-9_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking cpp-9 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package cpp.
Preparing to unpack .../20-cpp_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking cpp (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package libcc1-0:amd64.
Preparing to unpack .../21-libcc1-0_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libcc1-0:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libgomp1:amd64.
Preparing to unpack .../22-libgomp1_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libgomp1:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libitm1:amd64.
Preparing to unpack .../23-libitm1_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libitm1:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../24-libatomic1_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libatomic1:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libasan5:amd64.
Preparing to unpack .../25-libasan5_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking libasan5:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../26-liblsan0_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking liblsan0:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libtsan0:amd64.
Preparing to unpack .../27-libtsan0_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libtsan0:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libubsan1:amd64.
Preparing to unpack .../28-libubsan1_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libubsan1:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libquadmath0:amd64.
Preparing to unpack .../29-libquadmath0_10.5.0-1ubuntu1~20.04_amd64.deb ...
Unpacking libquadmath0:amd64 (10.5.0-1ubuntu1~20.04) ...
Selecting previously unselected package libgcc-9-dev:amd64.
Preparing to unpack .../30-libgcc-9-dev_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking libgcc-9-dev:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package gcc-9.
Preparing to unpack .../31-gcc-9_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking gcc-9 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package gcc.
Preparing to unpack .../32-gcc_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking gcc (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package libstdc++-9-dev:amd64.
Preparing to unpack .../33-libstdc++-9-dev_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking libstdc++-9-dev:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package g++-9.
Preparing to unpack .../34-g++-9_9.4.0-1ubuntu1~20.04.2_amd64.deb ...
Unpacking g++-9 (9.4.0-1ubuntu1~20.04.2) ...
Selecting previously unselected package g++.
Preparing to unpack .../35-g++_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking g++ (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package make.
Preparing to unpack .../36-make_4.2.1-1.2_amd64.deb ...
Unpacking make (4.2.1-1.2) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../37-libdpkg-perl_1.19.7ubuntu3.2_all.deb ...
Unpacking libdpkg-perl (1.19.7ubuntu3.2) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../38-dpkg-dev_1.19.7ubuntu3.2_all.deb ...
Unpacking dpkg-dev (1.19.7ubuntu3.2) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../39-build-essential_12.8ubuntu1.1_amd64.deb ...
Unpacking build-essential (12.8ubuntu1.1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../40-containerd.io_1.6.24-1_amd64.deb ...
Unpacking containerd.io (1.6.24-1) ...
Selecting previously unselected package docker-buildx-plugin.
Preparing to unpack .../41-docker-buildx-plugin_0.11.2-1~ubuntu.20.04~focal_amd64.deb ...
Unpacking docker-buildx-plugin (0.11.2-1~ubuntu.20.04~focal) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../42-docker-ce-cli_5%3a24.0.6-1~ubuntu.20.04~focal_amd64.deb ...
Unpacking docker-ce-cli (5:24.0.6-1~ubuntu.20.04~focal) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../43-docker-ce_5%3a24.0.6-1~ubuntu.20.04~focal_amd64.deb ...
Unpacking docker-ce (5:24.0.6-1~ubuntu.20.04~focal) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../44-docker-ce-rootless-extras_5%3a24.0.6-1~ubuntu.20.04~focal_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:24.0.6-1~ubuntu.20.04~focal) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../45-docker-compose-plugin_2.21.0-1~ubuntu.20.04~focal_amd64.deb ...
Unpacking docker-compose-plugin (2.21.0-1~ubuntu.20.04~focal) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../46-libfakeroot_1.24-1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.24-1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../47-fakeroot_1.24-1_amd64.deb ...
Unpacking fakeroot (1.24-1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../48-libalgorithm-diff-perl_1.19.03-2_all.deb ...
Unpacking libalgorithm-diff-perl (1.19.03-2) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../49-libalgorithm-diff-xs-perl_0.04-6_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-6) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../50-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../51-libexpat1-dev_2.2.9-1ubuntu0.6_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.9-1ubuntu0.6) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../52-libfile-fcntllock-perl_0.22-3build4_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-3build4) ...
Selecting previously unselected package libpython3.8-dev:amd64.
Preparing to unpack .../53-libpython3.8-dev_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04.8) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../54-libpython3-dev_3.8.2-0ubuntu2_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.8.2-0ubuntu2) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../55-manpages-dev_5.05-1_all.deb ...
Unpacking manpages-dev (5.05-1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../56-python-pip-whl_20.0.2-5ubuntu1.9_all.deb ...
Unpacking python-pip-whl (20.0.2-5ubuntu1.9) ...
Selecting previously unselected package zlib1g-dev:amd64.
Preparing to unpack .../57-zlib1g-dev_1%3a1.2.11.dfsg-2ubuntu1.5_amd64.deb ...
Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.5) ...
Selecting previously unselected package python3.8-dev.
Preparing to unpack .../58-python3.8-dev_3.8.10-0ubuntu1~20.04.8_amd64.deb ...
Unpacking python3.8-dev (3.8.10-0ubuntu1~20.04.8) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../59-python3-dev_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3-dev (3.8.2-0ubuntu2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../60-python3-wheel_0.34.2-1ubuntu0.1_all.deb ...
Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../61-python3-pip_20.0.2-5ubuntu1.9_all.deb ...
Unpacking python3-pip (20.0.2-5ubuntu1.9) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../62-slirp4netns_0.4.3-1_amd64.deb ...
Unpacking slirp4netns (0.4.3-1) ...
Selecting previously unselected package unzip.
Preparing to unpack .../63-unzip_6.0-25ubuntu1.1_amd64.deb ...
Unpacking unzip (6.0-25ubuntu1.1) ...
Setting up manpages-dev (5.05-1) ...
Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.8) ...
Setting up slirp4netns (0.4.3-1) ...
Setting up libfile-fcntllock-perl (0.22-3build4) ...
Setting up libalgorithm-diff-perl (1.19.03-2) ...
Setting up unzip (6.0-25ubuntu1.1) ...
Setting up binutils-common:amd64 (2.34-6ubuntu1.6) ...
Setting up linux-libc-dev:amd64 (5.4.0-165.182) ...
Setting up libctf-nobfd0:amd64 (2.34-6ubuntu1.6) ...
Setting up libgomp1:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up python3-wheel (0.34.2-1ubuntu0.1) ...
Setting up libfakeroot:amd64 (1.24-1) ...
Setting up fakeroot (1.24-1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up docker-buildx-plugin (0.11.2-1~ubuntu.20.04~focal) ...
Setting up make (4.2.1-1.2) ...
Setting up libquadmath0:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up libmpc3:amd64 (1.1.0-1) ...
Setting up libatomic1:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up containerd.io (1.6.24-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up docker-compose-plugin (2.21.0-1~ubuntu.20.04~focal) ...
Setting up libdpkg-perl (1.19.7ubuntu3.2) ...
Setting up libubsan1:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up docker-ce-cli (5:24.0.6-1~ubuntu.20.04~focal) ...
Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04.8) ...
Setting up pigz (2.4-1) ...
Setting up libcrypt-dev:amd64 (1:4.4.10-10ubuntu4) ...
Setting up libisl22:amd64 (0.22.1-1) ...
Setting up python-pip-whl (20.0.2-5ubuntu1.9) ...
Setting up docker-ce-rootless-extras (5:24.0.6-1~ubuntu.20.04~focal) ...
Setting up libbinutils:amd64 (2.34-6ubuntu1.6) ...
Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.8) ...
Setting up libc-dev-bin (2.31-0ubuntu9.12) ...
Setting up python3.8 (3.8.10-0ubuntu1~20.04.8) ...
Setting up libalgorithm-diff-xs-perl (0.04-6) ...
Setting up libcc1-0:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up liblsan0:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up libitm1:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up gcc-9-base:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up libtsan0:amd64 (10.5.0-1ubuntu1~20.04) ...
Setting up libctf0:amd64 (2.34-6ubuntu1.6) ...
Setting up libasan5:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Setting up docker-ce (5:24.0.6-1~ubuntu.20.04~focal) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.8) ...
Setting up python3-pip (20.0.2-5ubuntu1.9) ...
Setting up cpp-9 (9.4.0-1ubuntu1~20.04.2) ...
Setting up libc6-dev:amd64 (2.31-0ubuntu9.12) ...
Setting up binutils-x86-64-linux-gnu (2.34-6ubuntu1.6) ...
Setting up binutils (2.34-6ubuntu1.6) ...
Setting up dpkg-dev (1.19.7ubuntu3.2) ...
Setting up libgcc-9-dev:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Setting up libexpat1-dev:amd64 (2.2.9-1ubuntu0.6) ...
Setting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04.8) ...
Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.5) ...
Setting up cpp (4:9.3.0-1ubuntu2) ...
Setting up gcc-9 (9.4.0-1ubuntu1~20.04.2) ...
Setting up libpython3-dev:amd64 (3.8.2-0ubuntu2) ...
Setting up libstdc++-9-dev:amd64 (9.4.0-1ubuntu1~20.04.2) ...
Setting up gcc (4:9.3.0-1ubuntu2) ...
Setting up g++-9 (9.4.0-1ubuntu1~20.04.2) ...
Setting up python3.8-dev (3.8.10-0ubuntu1~20.04.8) ...
Setting up g++ (4:9.3.0-1ubuntu2) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.8ubuntu1.1) ...
Setting up python3-dev (3.8.2-0ubuntu2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.21) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for mime-support (3.64ubuntu1) ...
--> Checking internet connection
--> Installing Lithops python dependencies
Collecting flask
  Downloading flask-3.0.0-py3-none-any.whl (99 kB)
Collecting gevent
  Downloading gevent-23.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
Collecting lithops[all]
  Downloading lithops-3.0.1-py3-none-any.whl (359 kB)
Collecting itsdangerous>=2.1.2
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting click>=8.1.3
  Downloading click-8.1.7-py3-none-any.whl (97 kB)
Collecting Werkzeug>=3.0.0
  Downloading werkzeug-3.0.1-py3-none-any.whl (226 kB)
Collecting blinker>=1.6.2
  Downloading blinker-1.6.3-py3-none-any.whl (13 kB)
Collecting Jinja2>=3.1.2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting importlib-metadata>=3.6.0; python_version < "3.10"
  Downloading importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting greenlet>=2.0.0; platform_python_implementation == "CPython" and python_version < "3.11"
  Downloading greenlet-3.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (662 kB)
Requirement already satisfied, skipping upgrade: zope.interface in /usr/lib/python3/dist-packages (from gevent) (4.7.1)
Collecting zope.event
  Downloading zope.event-5.0-py3-none-any.whl (6.8 kB)
Collecting matplotlib
  Downloading matplotlib-3.7.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
Collecting pika
  Downloading pika-1.3.2-py3-none-any.whl (155 kB)
Collecting seaborn
  Downloading seaborn-0.13.0-py3-none-any.whl (294 kB)
Collecting redis
  Downloading redis-5.0.1-py3-none-any.whl (250 kB)
Collecting python-dateutil
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting cloudpickle
  Downloading cloudpickle-3.0.0-py3-none-any.whl (20 kB)
Collecting ibm-vpc
  Downloading ibm-vpc-0.19.1.tar.gz (366 kB)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/lib/python3/dist-packages (from lithops[all]) (5.3.1)
Collecting ibm-code-engine-sdk
  Downloading ibm-code-engine-sdk-3.1.0.tar.gz (47 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied, skipping upgrade: requests in /usr/lib/python3/dist-packages (from lithops[all]) (2.22.0)
Collecting paramiko
  Downloading paramiko-3.3.1-py3-none-any.whl (224 kB)
Collecting docker
  Downloading docker-6.1.3-py3-none-any.whl (148 kB)
Collecting ps-mem
  Downloading ps_mem-3.14-py2.py3-none-any.whl (19 kB)
Collecting lxml
  Downloading lxml-4.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.1 MB)
Collecting tblib
  Downloading tblib-3.0.0-py3-none-any.whl (12 kB)
Collecting tqdm
  Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
Collecting pandas
  Downloading pandas-2.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)
Collecting kubernetes
  Downloading kubernetes-28.1.0-py2.py3-none-any.whl (1.6 MB)
Collecting ibm-cos-sdk
  Downloading ibm-cos-sdk-2.13.2.tar.gz (56 kB)
Collecting boto3
  Downloading boto3-1.28.70-py3-none-any.whl (135 kB)
Collecting azure-mgmt-network; extra == "all"
  Downloading azure_mgmt_network-25.1.0-py3-none-any.whl (656 kB)
Collecting diskcache; extra == "all"
  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
Collecting azure-storage-queue; extra == "all"
  Downloading azure_storage_queue-12.7.3-py3-none-any.whl (173 kB)
Collecting azure-mgmt-compute; extra == "all"
  Downloading azure_mgmt_compute-30.3.0-py3-none-any.whl (5.2 MB)
Collecting joblib; extra == "all"
  Downloading joblib-1.3.2-py3-none-any.whl (302 kB)
Collecting aliyun-fc2; extra == "all"
  Downloading aliyun-fc2-2.5.2.tar.gz (16 kB)
Collecting google-api-python-client; extra == "all"
  Downloading google_api_python_client-2.105.0-py2.py3-none-any.whl (12.6 MB)
Collecting azure-mgmt-resource; extra == "all"
  Downloading azure_mgmt_resource-23.0.1-py3-none-any.whl (2.5 MB)
Collecting google-auth; extra == "all"
  Downloading google_auth-2.23.3-py2.py3-none-any.whl (182 kB)
Collecting google-cloud-pubsub; extra == "all"
  Downloading google_cloud_pubsub-2.18.4-py2.py3-none-any.whl (265 kB)
Collecting oss2; extra == "all"
  Downloading oss2-2.18.2.tar.gz (273 kB)
Requirement already satisfied, skipping upgrade: httplib2; extra == "all" in /usr/lib/python3/dist-packages (from lithops[all]) (0.14.0)
Collecting numpy; extra == "all"
  Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting azure-identity; extra == "all"
  Downloading azure_identity-1.14.1-py3-none-any.whl (161 kB)
Collecting oci; extra == "all"
  Downloading oci-2.114.0-py3-none-any.whl (23.7 MB)
Collecting google-cloud-storage; extra == "all"
  Downloading google_cloud_storage-2.12.0-py2.py3-none-any.whl (120 kB)
Collecting pynng; extra == "all"
  Downloading pynng-0.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (889 kB)
Collecting azure-storage-blob; extra == "all"
  Downloading azure_storage_blob-12.18.3-py3-none-any.whl (392 kB)
Collecting MarkupSafe>=2.1.1
  Downloading MarkupSafe-2.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/lib/python3/dist-packages (from importlib-metadata>=3.6.0; python_version < "3.10"->flask) (1.0.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from zope.event->gevent) (45.2.0)
Collecting pillow>=6.2.0
  Downloading Pillow-10.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)
Collecting importlib-resources>=3.2.0; python_version < "3.10"
  Downloading importlib_resources-6.1.0-py3-none-any.whl (33 kB)
Collecting contourpy>=1.0.1
  Downloading contourpy-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)
Collecting pyparsing>=2.3.1
  Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.5-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting packaging>=20.0
  Downloading packaging-23.2-py3-none-any.whl (53 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.43.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Collecting cycler>=0.10
  Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
Collecting async-timeout>=4.0.2; python_full_version <= "3.11.2"
  Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->lithops[all]) (1.14.0)
Collecting ibm_cloud_sdk_core>=3.16.7
  Downloading ibm-cloud-sdk-core-3.17.2.tar.gz (55 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting urllib3<2.0.0,>=1.26.0
  Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
Collecting bcrypt>=3.2
  Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (593 kB)
Collecting pynacl>=1.5
  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
Collecting cryptography>=3.3
  Downloading cryptography-41.0.5-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB)
Collecting websocket-client>=0.32.0
  Downloading websocket_client-1.6.4-py3-none-any.whl (57 kB)
Collecting tzdata>=2022.1
  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting pytz>=2020.1
  Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Requirement already satisfied, skipping upgrade: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->lithops[all]) (2019.11.28)
Collecting requests-oauthlib
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting oauthlib>=3.2.2
  Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
Collecting ibm-cos-sdk-core==2.13.2
  Downloading ibm-cos-sdk-core-2.13.2.tar.gz (1.1 MB)
Collecting ibm-cos-sdk-s3transfer==2.13.2
  Downloading ibm-cos-sdk-s3transfer-2.13.2.tar.gz (139 kB)
Collecting jmespath<=1.0.1,>=0.10.0
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting s3transfer<0.8.0,>=0.7.0
  Downloading s3transfer-0.7.0-py3-none-any.whl (79 kB)
Collecting botocore<1.32.0,>=1.31.70
  Downloading botocore-1.31.70-py3-none-any.whl (11.3 MB)
Collecting isodate<1.0.0,>=0.6.1
  Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting azure-common~=1.1
  Downloading azure_common-1.1.28-py2.py3-none-any.whl (14 kB)
Collecting azure-mgmt-core<2.0.0,>=1.3.2
  Downloading azure_mgmt_core-1.4.0-py3-none-any.whl (27 kB)
Collecting typing-extensions>=4.3.0
  Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting azure-core<2.0.0,>=1.28.0
  Downloading azure_core-1.29.5-py3-none-any.whl (192 kB)
Collecting google-auth-httplib2>=0.1.0
  Downloading google_auth_httplib2-0.1.1-py2.py3-none-any.whl (9.3 kB)
Collecting uritemplate<5,>=3.0.1
  Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0.dev0,>=1.31.5
  Downloading google_api_core-2.12.0-py3-none-any.whl (121 kB)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.3.2-py3-none-any.whl (9.3 kB)
Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth; extra == "all"->lithops[all]) (0.2.1)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting grpcio-status>=1.33.2
  Downloading grpcio_status-1.59.0-py3-none-any.whl (14 kB)
Collecting grpcio<2.0dev,>=1.51.3
  Downloading grpcio-1.59.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
Collecting grpc-google-iam-v1<1.0.0dev,>=0.12.4
  Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
  Downloading protobuf-4.24.4-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
Collecting proto-plus<2.0.0dev,>=1.22.0
  Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
Collecting aliyun-python-sdk-core>=2.13.12
  Downloading aliyun-python-sdk-core-2.14.0.tar.gz (443 kB)
Collecting aliyun-python-sdk-kms>=2.4.1
  Downloading aliyun_python_sdk_kms-2.16.2-py2.py3-none-any.whl (94 kB)
Collecting crcmod>=1.7
  Downloading crcmod-1.7.tar.gz (89 kB)
Collecting pycryptodome>=3.4.7
  Downloading pycryptodome-3.19.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting msal-extensions<2.0.0,>=0.3.0
  Downloading msal_extensions-1.0.0-py2.py3-none-any.whl (19 kB)
Collecting msal<2.0.0,>=1.20.0
  Downloading msal-1.24.1-py2.py3-none-any.whl (95 kB)
Requirement already satisfied, skipping upgrade: pyOpenSSL<24.0.0,>=17.5.0 in /usr/lib/python3/dist-packages (from oci; extra == "all"->lithops[all]) (19.0.0)
Collecting circuitbreaker<2.0.0,>=1.3.1
  Downloading circuitbreaker-1.4.0.tar.gz (9.7 kB)
Collecting google-crc32c<2.0dev,>=1.0
  Downloading google_crc32c-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Collecting google-resumable-media>=2.6.0
  Downloading google_resumable_media-2.6.0-py2.py3-none-any.whl (80 kB)
Collecting google-cloud-core<3.0dev,>=2.3.0
  Downloading google_cloud_core-2.3.3-py2.py3-none-any.whl (29 kB)
Collecting cffi
  Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (444 kB)
Collecting sniffio
  Downloading sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting PyJWT<3.0.0,>=2.8.0
  Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB)
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2
  Downloading googleapis_common_protos-1.61.0-py2.py3-none-any.whl (230 kB)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4->google-auth; extra == "all"->lithops[all]) (0.4.2)
Collecting portalocker<3,>=1.0; python_version >= "3.5" and platform_system != "Windows"
  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: ibm-vpc, ibm-code-engine-sdk, ibm-cos-sdk, aliyun-fc2, oss2, ibm-cloud-sdk-core, ibm-cos-sdk-core, ibm-cos-sdk-s3transfer, aliyun-python-sdk-core, crcmod, circuitbreaker
  Building wheel for ibm-vpc (setup.py): started
  Building wheel for ibm-vpc (setup.py): finished with status 'done'
  Created wheel for ibm-vpc: filename=ibm_vpc-0.19.1-py3-none-any.whl size=374070 sha256=149f914519cbca5057f9a1ff51424b92f7224e14a2cdc8a3bc85a221c62bec51
  Stored in directory: /root/.cache/pip/wheels/f3/03/f0/2ae9d44409785233203acc71a561eff1bd7b64223344bc6ead
  Building wheel for ibm-code-engine-sdk (PEP 517): started
  Building wheel for ibm-code-engine-sdk (PEP 517): finished with status 'done'
  Created wheel for ibm-code-engine-sdk: filename=ibm_code_engine_sdk-3.1.0-py3-none-any.whl size=48747 sha256=9444749c2d0cef9ebb3e74b5e9f4c1e3dbb5f8310bea80bebb2be338f7de6712
  Stored in directory: /root/.cache/pip/wheels/9e/18/54/f33d814345213d5a39dca7e999295fb827653bf2ed80f7ccf3
  Building wheel for ibm-cos-sdk (setup.py): started
  Building wheel for ibm-cos-sdk (setup.py): finished with status 'done'
  Created wheel for ibm-cos-sdk: filename=ibm_cos_sdk-2.13.2-py3-none-any.whl size=73917 sha256=27bdb40b95ed15ed6d8b72724c919989ebfe053d670b1bd548f4bbba1e45f674
  Stored in directory: /root/.cache/pip/wheels/0c/9c/19/4229e2f9709475e9ab6a7a9639d9309d550ee9f8d882852982
  Building wheel for aliyun-fc2 (setup.py): started
  Building wheel for aliyun-fc2 (setup.py): finished with status 'done'
  Created wheel for aliyun-fc2: filename=aliyun_fc2-2.5.2-py3-none-any.whl size=16995 sha256=343ca3ee0f43ec88671cba8434250498854edf0ac46972f12b967a25a1d8ae6b
  Stored in directory: /root/.cache/pip/wheels/62/a5/d9/60ebfcb7b77c655e5f8e2003bea2731b0b7ba16a4c01f62787
  Building wheel for oss2 (setup.py): started
  Building wheel for oss2 (setup.py): finished with status 'done'
  Created wheel for oss2: filename=oss2-2.18.2-py3-none-any.whl size=115312 sha256=1b4cff87b0b573b63c26e24d50e935030699c8c2848ed2076c41312b87d350a3
  Stored in directory: /root/.cache/pip/wheels/9b/90/df/9bfd98540652cc133eb71afda31c694e584142f68bb9a5d302
  Building wheel for ibm-cloud-sdk-core (PEP 517): started
  Building wheel for ibm-cloud-sdk-core (PEP 517): finished with status 'done'
  Created wheel for ibm-cloud-sdk-core: filename=ibm_cloud_sdk_core-3.17.2-py3-none-any.whl size=87033 sha256=0f6e3f76f5a9e10ffb71316bd300b87c891c0ba1e46e999e07e9a534478bb682
  Stored in directory: /root/.cache/pip/wheels/a9/c1/a7/58c9f3878b67b11f1192188000543bae9cc1bcdce1d65f956c
  Building wheel for ibm-cos-sdk-core (setup.py): started
  Building wheel for ibm-cos-sdk-core (setup.py): finished with status 'done'
  Created wheel for ibm-cos-sdk-core: filename=ibm_cos_sdk_core-2.13.2-py3-none-any.whl size=634575 sha256=94e58a13822fd7ef06646845877cbd5c8eff11ebecca2e5a9079ab8a3fcedd16
  Stored in directory: /root/.cache/pip/wheels/df/03/a2/1fc3f3a1710d97d2de18e73c535fcf535148a3adece021c626
  Building wheel for ibm-cos-sdk-s3transfer (setup.py): started
  Building wheel for ibm-cos-sdk-s3transfer (setup.py): finished with status 'done'
  Created wheel for ibm-cos-sdk-s3transfer: filename=ibm_cos_sdk_s3transfer-2.13.2-py3-none-any.whl size=90103 sha256=6b18f5262993ccfc83669f8a624017f2ac80437ed06a9a7e55ac22222c8c8ce1
  Stored in directory: /root/.cache/pip/wheels/0e/b0/d0/cb1c9f12d546dda15cd15f759ee8b0902918583255926477f6
  Building wheel for aliyun-python-sdk-core (setup.py): started
  Building wheel for aliyun-python-sdk-core (setup.py): finished with status 'done'
  Created wheel for aliyun-python-sdk-core: filename=aliyun_python_sdk_core-2.14.0-py3-none-any.whl size=535282 sha256=a55a1ac664d87982fd1226277bdc098a09035ce58770603076679db4d56aebdc
  Stored in directory: /root/.cache/pip/wheels/d9/cb/a4/4faa1c591d80410fca7aa5009264ebcaeba247cc842555fc07
  Building wheel for crcmod (setup.py): started
  Building wheel for crcmod (setup.py): finished with status 'done'
  Created wheel for crcmod: filename=crcmod-1.7-cp38-cp38-linux_x86_64.whl size=35990 sha256=56355f50455227b679f21e438ee2994a37c01aab6eb116da28a16a4a613c1c65
  Stored in directory: /root/.cache/pip/wheels/ca/5a/02/f3acf982a026f3319fb3e798a8dca2d48fafee7761788562e9
  Building wheel for circuitbreaker (setup.py): started
  Building wheel for circuitbreaker (setup.py): finished with status 'done'
  Created wheel for circuitbreaker: filename=circuitbreaker-1.4.0-py3-none-any.whl size=7503 sha256=f34800eb4c3f87e6abb506a36d1b4f79dc0ea6dfb9ab58fb7adea45c4fab1454
  Stored in directory: /root/.cache/pip/wheels/52/06/a4/3f1cdbebf72c6da0c6e0074d5f4f54bebfa6c479ca6cb56e9d
Successfully built ibm-vpc ibm-code-engine-sdk ibm-cos-sdk aliyun-fc2 oss2 ibm-cloud-sdk-core ibm-cos-sdk-core ibm-cos-sdk-s3transfer aliyun-python-sdk-core crcmod circuitbreaker
ERROR: importlib-resources 6.1.0 has requirement zipp>=3.1.0; python_version < "3.10", but you'll have zipp 1.0.0 which is incompatible.
ERROR: ibm-cloud-sdk-core 3.17.2 has requirement requests<3.0.0,>=2.31.0, but you'll have requests 2.22.0 which is incompatible.
ERROR: ibm-code-engine-sdk 3.1.0 has requirement requests<3.0.0,>=2.26.0, but you'll have requests 2.22.0 which is incompatible.
ERROR: docker 6.1.3 has requirement requests>=2.26.0, but you'll have requests 2.22.0 which is incompatible.
ERROR: kubernetes 28.1.0 has requirement pyyaml>=5.4.1, but you'll have pyyaml 5.3.1 which is incompatible.
ERROR: ibm-cos-sdk-core 2.13.2 has requirement requests<3.0,>=2.31.0, but you'll have requests 2.22.0 which is incompatible.
ERROR: google-auth-httplib2 0.1.1 has requirement httplib2>=0.19.0, but you'll have httplib2 0.14.0 which is incompatible.
ERROR: google-api-python-client 2.105.0 has requirement httplib2<1.dev0,>=0.15.0, but you'll have httplib2 0.14.0 which is incompatible.
ERROR: aliyun-python-sdk-core 2.14.0 has requirement jmespath<1.0.0,>=0.9.3, but you'll have jmespath 1.0.1 which is incompatible.
Installing collected packages: itsdangerous, click, MarkupSafe, Werkzeug, blinker, Jinja2, importlib-metadata, flask, greenlet, zope.event, gevent, numpy, pillow, importlib-resources, contourpy, pyparsing, kiwisolver, packaging, fonttools, python-dateutil, cycler, matplotlib, pika, tzdata, pytz, pandas, seaborn, async-timeout, redis, cloudpickle, urllib3, PyJWT, ibm-cloud-sdk-core, ibm-vpc, ibm-code-engine-sdk, bcrypt, pycparser, cffi, pynacl, cryptography, paramiko, websocket-client, docker, ps-mem, lxml, tblib, tqdm, cachetools, rsa, google-auth, oauthlib, requests-oauthlib, kubernetes, jmespath, ibm-cos-sdk-core, ibm-cos-sdk-s3transfer, ibm-cos-sdk, botocore, s3transfer, boto3, isodate, azure-common, typing-extensions, azure-core, azure-mgmt-core, azure-mgmt-network, diskcache, azure-storage-queue, azure-mgmt-compute, joblib, aliyun-fc2, google-auth-httplib2, uritemplate, protobuf, googleapis-common-protos, google-api-core, google-api-python-client, azure-mgmt-resource, grpcio, grpcio-status, grpc-google-iam-v1, proto-plus, google-cloud-pubsub, aliyun-python-sdk-core, aliyun-python-sdk-kms, crcmod, pycryptodome, oss2, msal, portalocker, msal-extensions, azure-identity, circuitbreaker, oci, google-crc32c, google-resumable-media, google-cloud-core, google-cloud-storage, sniffio, pynng, azure-storage-blob, lithops
  Attempting uninstall: click
    Found existing installation: Click 7.0
    Not uninstalling click at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'Click'. No files were found to uninstall.
  Attempting uninstall: MarkupSafe
    Found existing installation: MarkupSafe 1.1.0
    Not uninstalling markupsafe at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'MarkupSafe'. No files were found to uninstall.
  Attempting uninstall: blinker
    Found existing installation: blinker 1.4
    Not uninstalling blinker at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'blinker'. No files were found to uninstall.
  Attempting uninstall: Jinja2
    Found existing installation: Jinja2 2.10.1
    Not uninstalling jinja2 at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'Jinja2'. No files were found to uninstall.
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 1.5.0
    Not uninstalling importlib-metadata at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'importlib-metadata'. No files were found to uninstall.
  Attempting uninstall: urllib3
    Found existing installation: urllib3 1.25.8
    Not uninstalling urllib3 at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'urllib3'. No files were found to uninstall.
  Attempting uninstall: PyJWT
    Found existing installation: PyJWT 1.7.1
    Not uninstalling pyjwt at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'PyJWT'. No files were found to uninstall.
  Attempting uninstall: pynacl
    Found existing installation: PyNaCl 1.3.0
    Not uninstalling pynacl at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'PyNaCl'. No files were found to uninstall.
  Attempting uninstall: cryptography
    Found existing installation: cryptography 2.8
    Not uninstalling cryptography at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'cryptography'. No files were found to uninstall.
  Attempting uninstall: oauthlib
    Found existing installation: oauthlib 3.1.0
    Not uninstalling oauthlib at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'oauthlib'. No files were found to uninstall.
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.3 PyJWT-2.8.0 Werkzeug-3.0.1 aliyun-fc2-2.5.2 aliyun-python-sdk-core-2.14.0 aliyun-python-sdk-kms-2.16.2 async-timeout-4.0.3 azure-common-1.1.28 azure-core-1.29.5 azure-identity-1.14.1 azure-mgmt-compute-30.3.0 azure-mgmt-core-1.4.0 azure-mgmt-network-25.1.0 azure-mgmt-resource-23.0.1 azure-storage-blob-12.18.3 azure-storage-queue-12.7.3 bcrypt-4.0.1 blinker-1.6.3 boto3-1.28.70 botocore-1.31.70 cachetools-5.3.2 cffi-1.16.0 circuitbreaker-1.4.0 click-8.1.7 cloudpickle-3.0.0 contourpy-1.1.1 crcmod-1.7 cryptography-41.0.5 cycler-0.12.1 diskcache-5.6.3 docker-6.1.3 flask-3.0.0 fonttools-4.43.1 gevent-23.9.1 google-api-core-2.12.0 google-api-python-client-2.105.0 google-auth-2.23.3 google-auth-httplib2-0.1.1 google-cloud-core-2.3.3 google-cloud-pubsub-2.18.4 google-cloud-storage-2.12.0 google-crc32c-1.5.0 google-resumable-media-2.6.0 googleapis-common-protos-1.61.0 greenlet-3.0.0 grpc-google-iam-v1-0.12.6 grpcio-1.59.0 grpcio-status-1.59.0 ibm-cloud-sdk-core-3.17.2 ibm-code-engine-sdk-3.1.0 ibm-cos-sdk-2.13.2 ibm-cos-sdk-core-2.13.2 ibm-cos-sdk-s3transfer-2.13.2 ibm-vpc-0.19.1 importlib-metadata-6.8.0 importlib-resources-6.1.0 isodate-0.6.1 itsdangerous-2.1.2 jmespath-1.0.1 joblib-1.3.2 kiwisolver-1.4.5 kubernetes-28.1.0 lithops-3.0.1 lxml-4.9.3 matplotlib-3.7.3 msal-1.24.1 msal-extensions-1.0.0 numpy-1.24.4 oauthlib-3.2.2 oci-2.114.0 oss2-2.18.2 packaging-23.2 pandas-2.0.3 paramiko-3.3.1 pika-1.3.2 pillow-10.1.0 portalocker-2.8.2 proto-plus-1.22.3 protobuf-4.24.4 ps-mem-3.14 pycparser-2.21 pycryptodome-3.19.0 pynacl-1.5.0 pynng-0.7.2 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 redis-5.0.1 requests-oauthlib-1.3.1 rsa-4.9 s3transfer-0.7.0 seaborn-0.13.0 sniffio-1.3.0 tblib-3.0.0 tqdm-4.66.1 typing-extensions-4.8.0 tzdata-2023.3 uritemplate-4.1.1 urllib3-1.26.18 websocket-client-1.6.4 zope.event-5.0
Created symlink /etc/systemd/system/multi-user.target.wants/lithops-master.service → /etc/systemd/system/lithops-master.service.
Generating public/private rsa key pair.
Your identification has been saved in /home/ubuntu/.ssh/id_rsa
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:BXKO3+GZCBt564Yc1B5dYwu6e/7l0ujLOICubveEHfQ root@ip-172-31-44-207
The key's randomart image is:
+---[RSA 3072]----+

After the installation fails, it turns out that pip is broken:

$ sudo pip list
Traceback (most recent call last):
  File "/usr/bin/pip", line 11, in <module>
    load_entry_point('pip==20.0.2', 'console_scripts', 'pip')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/usr/lib/python3/dist-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 65, in <module>
    vendored("cachecontrol")
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 36, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/adapter.py", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 12, in <module>
    import os, glob, subprocess, os.path, time, pwd, sys, requests_unixsocket
  File "/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py", line 1, in <module>
    import requests
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

Original exception was:
Traceback (most recent call last):
  File "/usr/bin/pip", line 11, in <module>
    load_entry_point('pip==20.0.2', 'console_scripts', 'pip')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/usr/lib/python3/dist-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 65, in <module>
    vendored("cachecontrol")
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 36, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/adapter.py", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1553, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1573, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'
JosepSampe commented 9 months ago

Based on this link it seems it is required to update the pyOpenSSL lib as well. I will try it, and include it in the installation script if needed

sergii-mamedov commented 9 months ago

I am using Ubuntu 20.04 with Python 3.8. In order for lithops to work on an EC2 instance, I install the following packages in this order:

sudo apt install python3-pip

sudo pip install PyYAML==5.4.1
sudo pip install requests==2.31.0
sudo pip install httplib2==0.19.0
sudo pip install urllib3==1.26.16
sudo pip install pyOpenSSL==23.2.0

sudo pip install flask gevent
sudo pip install lithops==3.0.1

Those packages for which the specified version is forced to be exactly that, because otherwise a conflict of versions occurs. After that I run my pipeline on another machine. Then lithops understands that it is necessary to install docker and auxiliary packages and everything seems to start working. Unfortunately I had another issue:

Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 334, in run
    return_vals = executor.get_result(futures)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 480, in get_result
    fs_done, _ = self.wait(
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 448, in wait
    raise e
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 423, in wait
    wait(fs=futures,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py", line 160, in wait
    raise e
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py", line 141, in wait
    new_data = _get_executor_data(fs, executor_data, pbar=pbar,
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py", line 285, in _get_executor_data
    list(pool.map(get_result, fs_to_wait_on))
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py", line 278, in get_result
    f.result(throw_except=throw_except, internal_storage=exec_data.internal_storage)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/future.py", line 324, in result
    self.status(throw_except=throw_except, internal_storage=internal_storage)
  File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/future.py", line 219, in status
    self._exception = pickle.loads(eval(self._call_status['exc_info']))
TypeError: unpickle_exception() takes 4 positional arguments but 7 were given
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-b7f9cd0b45e3> in <module>
----> 1 pipe.load_ds(use_cache=False)

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/cache.py in wrapper(self, *args, **kwargs)
     79             cacher = None
     80         if not cacher:
---> 81             return f(self, *args, **kwargs)
     82 
     83         if use_cache and cacher.exists(f_name):

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py in load_ds(self)
    138             self.ds_segm_lens,
    139             self.ds_size_hash,
--> 140         ) = load_ds(
    141             self.executor, self.imzml_cobject, self.ibd_cobject, self.ds_segm_size_mb, self.ds_id
    142         )

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/load_ds.py in load_ds(executor, imzml_cobject, ibd_cobject, ds_segm_size_mb, ds_id)
    286         imzml_browser_cobjs,
    287         md5_hash,
--> 288     ) = executor.call(
    289         _load_ds,
    290         (imzml_cobject, ibd_cobject, ds_segm_size_mb),

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py in call(self, func, args, runtime_memory, cost_factors, **kwargs)
    401             ), "Invalid cost_factors"
    402 
--> 403         return self.map(
    404             func, [args], runtime_memory=runtime_memory, cost_factors=cost_factors, **kwargs
    405         )[0]

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py in map(self, func, func_args, cost_factors, runtime_memory, include_modules, debug_run_locally, max_memory, **lithops_kwargs)
    296                     exc_info=exc,
    297                 )
--> 298                 raise exc
    299 
    300     def _dispatch_map(

/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py in run()
    332                             wrapper_func, func_args, runtime_memory=runtime_memory, **lithops_kwargs
    333                         )
--> 334                         return_vals = executor.get_result(futures)
    335                         if is_standalone:
    336                             # Dismantle & wait for it to stop while the mutex is still active

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py in get_result(self, fs, throw_except, timeout, threadpool_size, wait_dur_sec, show_progressbar)
    478         :return: The result of the future/s
    479         """
--> 480         fs_done, _ = self.wait(
    481             fs=fs,
    482             throw_except=throw_except,

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py in wait(self, fs, throw_except, return_when, download_results, timeout, threadpool_size, wait_dur_sec, show_progressbar)
    446                 self.compute_handler.clear(present_jobs)
    447                 self.clean(clean_cloudobjects=False, force=True)
--> 448             raise e
    449 
    450         if download_results:

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py in wait(self, fs, throw_except, return_when, download_results, timeout, threadpool_size, wait_dur_sec, show_progressbar)
    421         # Start waiting for results
    422         try:
--> 423             wait(fs=futures,
    424                  internal_storage=self.internal_storage,
    425                  job_monitor=self.job_monitor,

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py in wait(fs, internal_storage, job_monitor, throw_except, return_when, download_results, timeout, threadpool_size, wait_dur_sec, show_progressbar)
    158 
    159     except Exception as e:
--> 160         raise e
    161 
    162     finally:

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py in wait(fs, internal_storage, job_monitor, throw_except, return_when, download_results, timeout, threadpool_size, wait_dur_sec, show_progressbar)
    139             while not _check_done(fs, return_when, download_results):
    140                 for executor_data in executors_data:
--> 141                     new_data = _get_executor_data(fs, executor_data, pbar=pbar,
    142                                                   throw_except=throw_except,
    143                                                   download_results=download_results,

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py in _get_executor_data(fs, exec_data, download_results, throw_except, threadpool_size, pbar)
    283     pool = cf.ThreadPoolExecutor(max_workers=threadpool_size)
    284     if download_results:
--> 285         list(pool.map(get_result, fs_to_wait_on))
    286     else:
    287         list(pool.map(get_status, fs_to_wait_on))

/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py in result_iterator()
    609                     # Careful not to keep a reference to the popped future
    610                     if timeout is None:
--> 611                         yield fs.pop().result()
    612                     else:
    613                         yield fs.pop().result(end_time - time.monotonic())

/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

/opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/thread.py in run(self)
     55 
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/wait.py in get_result(f)
    276 
    277     def get_result(f):
--> 278         f.result(throw_except=throw_except, internal_storage=exec_data.internal_storage)
    279 
    280     def get_status(f):

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/future.py in result(self, throw_except, internal_storage)
    322             internal_storage = InternalStorage(storage_config=self._storage_config)
    323 
--> 324         self.status(throw_except=throw_except, internal_storage=internal_storage)
    325 
    326         if self.done:

/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/future.py in status(self, throw_except, internal_storage, check_only)
    217         if self._call_status['exception']:
    218             self._set_state(ResponseFuture.State.Error)
--> 219             self._exception = pickle.loads(eval(self._call_status['exc_info']))
    220 
    221             msg1 = ('ExecutorID {} | JobID {} - There was an exception - Activation '

TypeError: unpickle_exception() takes 4 positional arguments but 7 were given
JosepSampe commented 9 months ago

This exception is the same as #1179 . we are exploring what can cause this

sergii-mamedov commented 9 months ago

JFYI: I use the exact same pipeline with the same inputs for serverless (aws_lambda) without any problems.

JosepSampe commented 9 months ago

Regarding the issue you had installing packages, I ran some tests with an Ubuntu 22 VM image and the installation process works properly without the issue you experienced here.. I would like to do some tests with the same env as you, can you share the AWS region and the Ubuntu 20 VM image (AMI) ID you are using?

sergii-mamedov commented 9 months ago

I also tested Ubuntu 22.04. I didn't have any problems either. But I'm still using python 3.8 and Ubuntu 22.04 has python 3.10 installed by default. Because I was able to install all dependencies manually on EC2 (Ubuntu 20.04) this problem is not significant. Especially since I'm going to upgrade to python 3.10/3.11 soon. If only I encountered this - you can not waste time on this. More pressing for me is the unpickle_exception issue.

P.S.> region: eu-west-1 ami_id: ami-0136ddddd07f0584f ami-name: ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230517

sergii-mamedov commented 9 months ago

@JosepSampe I have a more general question about working with AWS EC2 + Docker because of one problem. I have two instances:

At the first launch on host_2, the lithops-master.service is created, which accepts requests from host_1 and performs the task. I'm using a Docker image to run the lithops executor code in an isolated environment (the image is on Docker Hub). I thought lithops-master.service should pull this image on host_2 and run it with docker. Currently I have an exception when running a simpler step of my pipeline that one of the python libraries is not available, even though it is definitely present in the docker image. So I have a feeling that my understanding of how running lithops executor on host_2 works is wrong. Although I use exactly the same approach as for IBM VPC.

sergii-mamedov commented 9 months ago

@JosepSampe The question is no longer relevant, I have solved the problem.

sergii-mamedov commented 8 months ago

@JosepSampe

  1. Is it necessary needs a compiler for aws_lambda docker image?
  2. I had a warning when I run on AWS Lambda. Unfortunately I can not find why it`s happens:
    2023-11-05 22:28:59,946 [DEBUG] jobrunner.py:203 -- Process started
    /usr/local/lib/python3.8/site-packages/joblib/_multiprocessing_helpers.py:46: UserWarning: [Errno 2] No such file or directory.  joblib will operate in serial mode
      warnings.warn('%s.  joblib will operate in serial mode' % (e,))
    2023-11-05 22:29:01,077 [INFO] jobrunner.py:233 -- Going to execute 'generate()'
JosepSampe commented 8 months ago

Is it necessary needs a compiler for aws_lambda docker image?

I think @aitorarjona added it here, so maybe he can provide a better answer than me on this. But in principle it is not required by Lithops, not sure if it is required by the awslambdaric lib for some reason.

I had a warning when I run on AWS Lambda. Unfortunately I can not find why it`s happens:

Would it be possible to share a piece of code to reproduce it?

sergii-mamedov commented 8 months ago

Would it be possible to share a piece of code to reproduce it?

This is part of a large pipeline. I will try to extract the input files. Could it be because AWS Lambda doesn't support shared memory for processes, which we talked about earlier?

JosepSampe commented 8 months ago

Yes, based on this thread it seems to be what you mention, since the error is produced in the _multiprocessing_helpers.py. Also mentioned in this other thread.

sergii-mamedov commented 8 months ago
  1. I turned on DEBUG logging and realized that serialization takes about 15 sec for all the dependencies at each step. At the same time, most of the steps take up to 10 seconds. Is there any way to do this serialization once for the first step? Or somehow add these dependencies inf the Docker image? As far as I understand, this happens in the SerializeIndependent and ModuleDependencyAnalyzer classes.
  2. Is it possible to specify normal short names for lamda function? Currently, the auto-generated entities are very long and have redundant entities. Examle: lithops_v3-0-1_lhdm__metaspace-aws-lambda--301c_512MB.
  3. Is it possible to turn off warnings for the code that is executed in the lithops executor?
  4. Is there a way to get a list of AWS RequestIds for each Lithops Executor ID? I see that the RequestId is in /tmp/lithops-ubuntu/functions.log but I would like to get it without parsing text files.
JosepSampe commented 8 months ago

I turned on DEBUG logging and realized that serialization takes about 15 sec for all the dependencies at each step. At the same time, most of the steps take up to 10 seconds. Is there any way to do this serialization once for the first step? Or somehow add these dependencies inf the Docker image? As far as I understand, this happens in the SerializeIndependent and ModuleDependencyAnalyzer classes.

Can you provide the debug log lines where you see this? I just want to see where exactly this is produced, and check if we can add an internal cache to prevent analyzing the modules again for subsequent maps in the same FunctionExecuor()

Is it possible to specify normal short names for lamda function? Currently, the auto-generated entities are very long and have redundant entities. Examle: lithops_v3-0-1_lhdm__metaspace-aws-lambda--301c_512MB.

This was originally designed like this because this was supporting some internal logic so that from the function name we were able to get the container name, the memory and the user. Not sure if this is still necessary, need to check. In other backends we switched from these kind of names to lithops-worker-xxxxxxx where xxxxx is the hash of the function name you see. Would this work for you? or what is for you a "normal short name"?

Is it possible to turn off warnings for the code that is executed in the lithops executor?

Which warning you mean? warnings from the boto3 lib? can you provide an example?

Is there a way to get a list of AWS RequestIds for each Lithops Executor ID? I see that the RequestId is in /tmp/lithops-ubuntu/functions.log but I would like to get it without parsing text files.

Do you mean this?

    fexec = lithops.FunctionExecutor(backend='aws_lambda', storage='aws_s3')
    futures = fexec.map(my_map_function, range(2))

    fexec.get_result()

    for fut in futures:
        print(fut.activation_id)
sergii-mamedov commented 8 months ago

Can you provide the debug log lines where you see this? I just want to see where exactly this is produced, and check if we can add an internal cache to prevent analyzing the modules again for subsequent maps in the same FunctionExecuor()

I have thousands of log lines for one run, here are the first couple dozen:

2023-11-06 17:24:41,579 - DEBUG - lithops.job.job[Thread-1-ex] - job.py:234 - ExecutorID 59a60d-2 | JobID M009 - Serializing function and data
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'scipy.sparse.csgraph._laplacian'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'scipy' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'sklearn._config'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'sklearn' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'scipy.linalg.basic'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'scipy' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'scipy.linalg.lapack'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'scipy' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'numpy.core.fromnumeric'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'numpy' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'itertools'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:117 - Inspecting module 'itertools'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:172 - Module 'itertools' is built-in. Skipping.
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'sklearn.manifold._spectral_embedding'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'sklearn' is to be ignored, skipping
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:66 - Queuing module 'sm.engine.image_storage'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:117 - Inspecting module 'sm'
2023-11-06 17:24:41,607 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:163 - Module 'sm' is package. Recursing...
2023-11-06 17:24:41,608 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:187 - Inspecting submodule 'engine'
2023-11-06 17:24:41,608 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:241 - 'sm' -> 'engine' is package. Recursing...
2023-11-06 17:24:41,608 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:187 - Inspecting submodule 'annotation'
2023-11-06 17:24:41,608 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:241 - 'sm' -> 'annotation' is package. Recursing...
2023-11-06 17:24:41,608 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:187 - Inspecting submodule 'acq_geometry'
2023-11-06 17:24:41,609 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:195 - 'sm' -> 'acq_geometry' is source/compiled. Scanning imports.
2023-11-06 17:24:41,609 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:211 - 'sm' -> 'acq_geometry' had these imports {'sm'}
2023-11-06 17:24:41,609 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:215 - 'sm' -> 'acq_geometry' -> 'sm' already inspected
2023-11-06 17:24:41,609 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:187 - Inspecting submodule 'diagnostics'
2023-11-06 17:24:41,609 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:195 - 'sm' -> 'diagnostics' is source/compiled. Scanning imports.
2023-11-06 17:24:41,611 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:211 - 'sm' -> 'diagnostics' had these imports {'io', 'enum', 'traceback', 'num
py', 'pandas', 'typing', 'datetime', 'logging', 'sm', 'collections'}
2023-11-06 17:24:41,611 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:225 - 'sm' -> 'diagnostics' -> 'io' to be ignored
2023-11-06 17:24:41,611 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:225 - 'sm' -> 'diagnostics' -> 'enum' to be ignored
2023-11-06 17:24:41,611 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:225 - 'sm' -> 'diagnostics' -> 'traceback' to be ignored
2023-11-06 17:24:41,611 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:225 - 'sm' -> 'diagnostics' -> 'numpy' to be ignored
JosepSampe commented 8 months ago

Could you share the last lines? these lines:

2023-11-13 19:31:12,678 [DEBUG] module_dependency.py:110 -- Module 'traceback' is to be ignored, skipping
2023-11-13 19:31:12,678 [DEBUG] serialize.py:84 -- Referenced modules: /home/josep/dev-workspace/personal/lithops/lithops/multiprocessing/process.py, /usr/lib/python3.10/os.py, /usr/lib/python3.10/concurrent/futures/thread.py, /usr/local/lib/python3.10/dist-packages/diskcache/__init__.py, /home/josep/dev-workspace/personal/lithops/lithops/util/joblib/lithops_backend.py, /usr/lib/python3.10/traceback.py, /home/josep/dev-workspace/personal/lithops/lithops/multiprocessing/util.py
2023-11-13 19:31:12,678 [DEBUG] serialize.py:101 -- Modules to transmit: /usr/local/lib/python3.10/dist-packages/diskcache
2023-11-13 19:31:12,680 [DEBUG] job.py:269 -- ExecutorID 923be9-0 | JobID M000 - Uploading function and modules to the storage backend
2023-11-13 19:31:13,194 [DEBUG] aws_s3.py:104 -- PUT Object lithops.jobs/923be9-0/9cc7060fae2b1e58b21541e3fe7cfffc.func.pickle - Size: 223.8KiB - OK
sergii-mamedov commented 8 months ago 
2023-11-06 17:24:29,319 - DEBUG - lithops.libs.multyvac.module_dependency[Thread-1-ex] - module_dependency.py:110 - Module 'pandas' is to be ignored, skipping
2023-11-06 17:24:29,320 - DEBUG - lithops.job.serialize[Thread-1-ex] - serialize.py:84 - Referenced modules: /opt/dev/metaspace/metaspace/engine/sm/engine/utils/perf_profile.py, /opt/dev/metaspace/metaspace/engine/sm/engine/image_storage.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/logging/__init__.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/concurrent/futures/thread.py, /opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/store_images.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pandas/core/series.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/pickle.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/lib-dynload/resource.cpython-38-x86_64-linux-gnu.so, /opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/collections/__init__.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pandas/__init__.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/storage/utils.py, /opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/io.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/datetime.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pyarrow/__init__.py
2023-11-06 17:24:29,320 - DEBUG - lithops.job.serialize[Thread-1-ex] - serialize.py:90 - Tentative modules to transmit: /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pylab.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/elasticsearch, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pluggy, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/jedi, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/jinja2, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pyasn1, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/ipython_genutils, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/ibm_s3transfer, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/prompt_toolkit, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/cycler.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/ptyprocess, /opt/dev/metaspace/metaspace/engine/sm, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/parso, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/iniconfig, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/ibm_boto3, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/bottle.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/yaml, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pyspark, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/rsa, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/typing_extensions.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/traitlets, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/patsy, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/ibm_botocore, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/decorator.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/fabric, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/wcwidth, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/tenacity, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pexpect, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pytest, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/backcall, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/_plotly_future_, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/py4j, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/_pytest, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/_plotly_utils, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/paramiko, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pysparkling, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/elastic_transport, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pygments, /opt/dev/metaspace/metaspace/engine/tests, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/metaspace, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/docutils, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/attr, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/pickleshare.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/packaging, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/png.py, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/IPython, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/cherrypy, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/seaborn, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/aiosignal, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/plotly, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/tqdm, /opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/boto
2023-11-06 17:24:29,320 - DEBUG - lithops.job.serialize[Thread-1-ex] - serialize.py:92 - Include modules: engine/sm
2023-11-06 17:24:29,320 - DEBUG - lithops.job.serialize[Thread-1-ex] - serialize.py:101 - Modules to transmit: /opt/dev/miniconda3/envs/sm38/lib/python3.8/lib-dynload/resource.cpython-38-x86_64-linux-gnu.so, /opt/dev/metaspace/metaspace/engine/sm
2023-11-06 17:24:29,325 - DEBUG - lithops.job.job[Thread-1-ex] - job.py:271 - ExecutorID 59a60d-2 | JobID M008 - Uploading function and modules to the storage backend
2023-11-06 17:24:29,439 - DEBUG - lithops.storage.backends.aws_s3.aws_s3[Thread-1-ex] - aws_s3.py:104 - PUT Object lithops.jobs/59a60d-2/7a2c251b537f01b1c84162a4ce6f581c.func.pickle - Size: 846.3KiB - OK
2023-11-06 17:24:29,439 - DEBUG - lithops.job.job[Thread-1-ex] - job.py:311 - ExecutorID 59a60d-2 | JobID M008 - Data per activation is < 8.0KiB. Passing data through invocation payload
JosepSampe commented 8 months ago

Thanks @sergii-mamedov , If I remember correctly, you are using the include_modules config parameter, right? regarding the lib lib-dynload/resource.cpython-38-x86_64-linux-gnu.so, did you include it in include_modules or it was automatically detected?

sergii-mamedov commented 8 months ago

I use include_modules in config file but only our local library (our code). lib-dynload/resource.cpython-38-x86_64-linux-gnu.so automatically detected

sergii-mamedov commented 8 months ago

Which warning you mean? warnings from the boto3 lib? can you provide an example?

I have a third-party library that warns about certain problems with the format of input data that is generated on the user side. Locally I can disable warning logging with python -no-warning foo.py. Can I disable warning logging so I don't see the following lines?

    ---------------------- FUNCTION LOG ----------------------
    /usr/local/lib/python3.8/site-packages/pyimzml/ontology/ontology.py:92: UserWarning: Accession IMS:1000491 found with incorrect name "line scan left right". Updating name to "linescan 
left right".
      warn(

Is it better to use this approach?

JosepSampe commented 8 months ago

Is it better to use this approach?

Yes, it looks good. Or alternatively you can try setting the os.environ['PYTHONWARNINGS'] = "ignore" at the beginning of the function code, or use the extra_env parameter in the call_async or map() call

sergii-mamedov commented 8 months ago

Do you mean this?

    fexec = lithops.FunctionExecutor(backend='aws_lambda', storage='aws_s3')
    futures = fexec.map(my_map_function, range(2))

    fexec.get_result()

    for fut in futures:
        print(fut.activation_id)

Yes, this is exactly what I need. I will ask just in case. Is there sometimes no built-in functionality to get the following information as well?

'REPORT RequestId: 74a72eb5-207c-484a-a759-1f9805adce9f\tDuration: 3437.72 ms\tBilled Duration: 3438 ms\tMemory Size: 2048 MB\tMax Memory Used: 379 MB\t\n'`

I do this by querying CloudWatch but it requires a bit of extra time and code.

JosepSampe commented 8 months ago

You can see the stats Lithops collects in here, but maybe what you get from cloudwatch is more accurate ( I don't know, I never compared them)

sergii-mamedov commented 8 months ago

You can see the stats Lithops collects in here, but maybe what you get from cloudwatch is more accurate ( I don't know, I never compared them)

I already use this data. They provide statistics only to the code that is in the middle of the block:

---------------------- FUNCTION LOG ----------------------
----------------------------------------------------------

Metrics from CloudWatch also take into account lambda initialization time, lithops initialization time inside lambda function. Therefore, these data are more accurate. But for now I can get them myself. Thanks for the answer.

JosepSampe commented 8 months ago

@sergii-mamedov FYI In #1199 I fixed the include_modules functionality so now you shouldn't experience those recurrent 15 seconds overhead In #1195 we changed the default mode in standalone. So if you use consume and you decide to update Lithops, you will have to include exec_mode: consume in your config

sergii-mamedov commented 8 months ago

@JosepSampe Tested last night. Much faster. Thanks.

JosepSampe commented 8 months ago

Closing this as the main issue was already solved. Feel free to open new issues as needed.