-
Why does Lithops always use the DEBUG level in these CLI commands and not the log level the user wants?
https://github.com/lithops-cloud/lithops/blob/436f458e0d862ca1aff35886eb03d5e2143896c9/lithop…
-
@cisaacstern and I were wondering what was the rationale of the lithops project making your own implementation for the lithops [cloud proxy storage API](https://lithops-cloud.github.io/docs/source/api…
-
Finding byte ranges in every file in an archival dataset is an embarrassingly-parallel problem, which might be a good fit for serverless.
This step is analogous to the `parallel=True` option to `xr…
-
Currently the S3Client needs the credentials to be used, but what happens if i already have a boto3 client somewhere initialized and i want to use the S3Client class?
Is it possible to allow for t…
-
The `process_centr_segments` step in Lithops annotation seems to take 3-4x as much CPU time as the Spark implementation for normal-sized datasets. For very small datasets the difference can be much bi…
-
I have the following code that counts words but I am getting an error on this line:
fhand = s3.open('lithops-data-yey/notsobigtextfile_small.txt')
I am not sure why it says forbidden and don't know …
-
VCF to Zarr conversion is an embarrassingly parallel process that currently uses Dask Delayed to schedule tasks.
It would be fairly easy to make it possible to run on any parallel backend - example…
-
Currently, Lithops can be configured:
- by a config YAML file in ~/.lithops/config
- by ENV vars
- by directly passing configuration path or dict to a FunctionExecutor constructor
The standar…
-
Could Spark be added as a supported executor?
Maybe RDD.map or RDD.mapPartitions would be the correct way to map a function similar to [`map_unordered`](https://github.com/cubed-dev/cubed/blob/main…
-
For very large computations when the number of tasks for an array is much greater than the number of workers, it may be desirable to have more control over task submission in the client, so a large nu…