Lithops 3.2.0

[sergii-mamedov]

Lithops upgrade

0. Setting up lithops config in ~/.lithops/config . For different environments (prod/staging), the configs should be different.
1. Change lithops version in requirements.txt
2. Commit this change and deploy on staging.
3. Run pip install -r requirements.txt on the local envs.
4. Select staging config and create docker image for Lambda and EC2.
5. For lambda function:
   1. Create docker image and upload to ECR: cd engine/docker/lithops_aws_lambda && ./build_and_push.sh 3.2.0.a
   2. On staging instance in config.json file change runtime to metaspace-aws-lambda:3.2.0.a in lithops -> aws_lambda section.
   3. On staging instance run manually load_ds step [1] for creating lambda function for different amount of RAM (512, 1024, 2048, 4096, 8192) MB.
   4. For each lambda function need to change CloudWatch log group to /aws/lambda/staging-lithops.
6. For EC2:
   1. Create docker image and upload to ECR: cd engine/docker/lithops_aws_ec2 && ./build_and_push.sh 3.2.0.a
   2. Create EC2 instances for different amount of RAM (32, 64, 128, 256) GB based on Ubuntu 22.04 and run this [2].
   3. On staging instance in config.json file change runtime to metaspace-aws-ec2:3.2.0.a in lithops -> aws_ec2 section and also instance_id and ec2_instances.
   4. On staging instance run manually load_ds step [1] for running lithops on each created EC instance (32, 64, 128, 256) GB.

[1] It is easiest to iteratively hardcode runtime memory after this block

from sm.engine.annotation_lithops.executor import Executor
from sm.engine.annotation_lithops.io import load_cobj, load_cobjs
from sm.engine.dataset import Dataset
from sm.engine.config import SMConfig
from sm.engine.utils.perf_profile import NullProfiler
from sm.engine.annotation_lithops.annotation_job import ServerAnnotationJob

from sm.engine.db import DB, ConnectionPool
from sm.engine.es_export import ESExporter
from sm.engine.util import init_loggers

config = SMConfig()
config.set_path('/opt/dev/metaspace/metaspace/engine/conf/config.json')
config = config.get_conf()
perf = NullProfiler()
executor = Executor(config['lithops'], perf)

init_loggers(config['logs'])
db = DB()
es = ESExporter(db)
connection_pool = ConnectionPool(config['db'])
connection_pool.__enter__()
globals().update(locals())

ds_id = '2023-09-11_13h55m06s'
job = ServerAnnotationJob(executor, Dataset.load(DB(), ds_id), perf, use_cache=False)
pipe = job.pipe
pipe.clean()

pipe.load_ds(use_cache=False)

[2]

sudo apt update
sudo apt install python3-pip -y

sudo pip install --upgrade blinker
sudo pip install --upgrade wheel
sudo pip install PyYAML==5.4.1
sudo pip install requests==2.31.0
sudo pip install httplib2==0.19.0
sudo pip install urllib3==1.26.16
sudo pip install pyOpenSSL==23.2.0
sudo pip install tblib==1.7.0
sudo pip install flask==2.3.3
sudo pip install redis==5.0.8
sudo pip install gevent