Bugs in 3.0.0

[sergii-mamedov]

We are upgrading from version 2.7.0 to 3.0.0. I found a few bugs during the update.

1. After runtime was removed when initializing the instance of the class https://github.com/lithops-cloud/lithops/pull/1140 this change has to be pushed through kwargs. For ServerlessExecutor, initialization in the form kwargs=kwargs creates an additional level of dictionary nesting and the following code block does not work correctly.
2. Here is a line of code that works incorrectly in the case of an exception with the following stacktrace:

   Traceback (most recent call last):
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/lithops.py", line 91, in _callback
   self._manager.annotate_lithops(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/dataset_manager.py", line 116, in annotate_lithops
   ServerAnnotationJob(executor, ds, perf, perform_enrichment=perform_enrichment).run()
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/annotation_job.py", line 347, in run
   self.results_dfs, self.png_cobjs, self.enrichment_data = self.pipe.run_pipeline(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py", line 104, in run_pipeline
   self.load_ds(use_cache=use_cache)
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/cache.py", line 81, in wrapper
   return f(self, *args, **kwargs)
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py", line 140, in load_ds
   ) = load_ds(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/load_ds.py", line 287, in load_ds
   ) = executor.call(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 402, in call
   return self.map(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 297, in map
   raise exc
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 330, in run
   futures = executor.map(
   TypeError: exceptions must derive from BaseException

3. After changes in this PR: https://github.com/lithops-cloud/lithops/pull/1077 it seems necessary to change the mod_paths variable to mods_paths (added s) here, here and here.
4. After changes in this PR: https://github.com/lithops-cloud/lithops/pull/1010 I have an exception:

   Traceback (most recent call last):
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/lithops.py", line 91, in _callback
   self._manager.annotate_lithops(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/dataset_manager.py", line 119, in annotate_lithops
   Colocalization(self._db).run_coloc_job_lithops(executor, ds, reprocess=del_first)
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/postprocessing/colocalization.py", line 442, in run_coloc_job_lithops
   job_cobjs = fexec.map_concat(
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 391, in map_concat
   results = self.map(func, args, runtime_memory=runtime_memory, **kwargs)
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 297, in map
   raise exc
   File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 330, in run
   futures = executor.map(
   File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/executors.py", line 255, in map
   job = create_map_job(
   File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/job/job.py", line 80, in create_map_job
   job = _create_job(
   File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/job/job.py", line 237, in _create_job
   func_and_data_ser, mod_paths = serializer([func] + iterdata, inc_modules, exc_modules)
   File "/opt/dev/miniconda3/envs/sm38/lib/python3.8/site-packages/lithops/job/serialize.py", line 85, in __call__
   direct_modules else ", ".join(direct_modules)))
   TypeError: sequence item 43: expected str instance, NoneType found

   The reason for this is that the sys module does not pass validation for this line of code.

[JosepSampe]

Thanks for reporting @sergii-mamedov

Regarding 1. I found the issue a few weeks ago and fixed it here Regarding 3 and 4, I just added a fix here

Regarding 2. Could you provide more information about the error you get that end up triggering raise ex? Do you get 409? 500? another error from the CE API? or is it not an ApiException?

[sergii-mamedov]

Regarding 2. I don't know for sure, it happened when the new runtime was deployed first time. I have seen this exception several times. I tried to add logging but this problem then disappeared.

Couldn't you release a new minor version with all fixes?

[JosepSampe]

In included a few changes in the CE backend that might help preventing the issue you exerienced #1161

[sergii-mamedov]

@JosepSampe I tested master branch today. Works well. @gilv Could you create a new release?

[theodev]

@gilv can you please release the new version?

[JosepSampe]

@sergii-mamedov @theodev We just released with @gilv the new version (unfortunately, he had no access to his email these weeks)

[theodev]

Thanks @JosepSampe and @gilv !

[sergii-mamedov]

Thanks a lot