search

metaspace2020 / Lithops-METASPACE

Lithops-based Serverless implementation of the METASPACE spatial metabolomics annotation pipeline
12 stars 4 forks source link

Trouble running experiment-1-typical.ipynb notebook #85

Open Cohen-J-Omer opened 3 years ago

Cohen-J-Omer commented 3 years ago

Hey guys, After following your instructions carefully, i was trying to run the the first notebook, as i encountered the following exception from the 5th cell (first cell under "Setup" ): "Exception: No such key /test-bucket-te/lithops.jobs/tmp/6fb066/cloudobject_1 found in storage." and progress stalled at 0/84.

I believe I've configured the lithops_config file appropriately (as I've been working on the Lithops project for the past couple of weeks, with no issues) and as instructed i've added to it the required fields stated under "Pure Serverless mode".

By the way i think you may want to consider updating the required version of cpyImagingMSpec in setup.py, as version 0.2.4 can't be downloaded.

I would greatly appreciate if you could assist me in the matter.

LachlanStuart commented 3 years ago

Hi @Cohen-J-Omer ,

I haven't seen that exception before. I have several hypotheses for what may have caused it.

The first hypothesis is that Lithops is cleaning up the CloudObjects generated by one of the functions before it can be used by a later function. Are you using Lithops 2.2.16 or a more recent version? I've only tested with 2.2.16, but I know in 2.2.17b1 they changed the behavior to do some cleaning after every job, so it's possible that there's a bug that cleans up CloudObjects even when they could still be used.

If you are on 2.2.16, it would be worth trying to set "data_cleaner": false in the Lithops config. I'm pretty sure I tested it with the default value ("data_cleaner": true), and want to ensure the code works with it enabled, but for diagnostic purposes it would be helpful to know if the data cleaner was the cause.

The second hypothesis is that the pipeline cache is holding references to a previous job that has already been cleaned up. You could work around this by changing the Pipeline() constructor to have use_db_cache=False, use_ds_cache=False so that it doesn't use the pipeline cache.

If neither of the above help, please post the full stack trace of the exception, as it may provide some more insight.

Regarding cpyImagingMSpec, we actually pinned that version to ensure the results were consistent with the production version of METASPACE, but unfortunately the older version doesn't have an OSX build. The newer version has a different algorithm which gives slightly different results. But yeah, this repository has is no longer used for validating the equivalence of the two implementations, so I'll look at updating that dependency for better compatibility.