lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Addition of ability to run Lithops on shared multi-user machines. #1014

Closed spiogit closed 1 year ago

spiogit commented 1 year ago

The current behaviour of Lithops is to create an unqualified subdirectory in /tmp for tracking various bits of information. When more than one user one a machine run a globally installed framework, this causes permission issues:

$ lithops test ... [INFO] config.py:131 -- Lithops v2.7.2.dev0 ... Traceback (most recent call last): ... PermissionError: [Errno 13] Permission denied: '/tmp/lithops/storage/lithops.jobs' ... Traceback (most recent call last): ... PermissionError: [Errno 13] Permission denied: '/tmp/lithops/cleaner/tmpkjj_17vq'

The patch fixes that by qualifying the directory with a username component. (Security versus comfort trade-off: Using the UID may be safer but is less readable than the $USER value.) Related to that, the patch also makes it possible to configure Lithops for all users without having to set LITHOPS_CONFIG_FILE, while still permitting them to override with their own configurations. The use of a default file in /etc follows conventions.

With this patch, Lithops becomes usable e.g. in classroom settings.

All but one unit tests pass. The one that bails out might be unrelated to the patch.

====================================================================== FAIL: test_storage_list_objects (lithops.tests.test_storage.TestStorage)

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/lithops-2.7.2.dev0-py3.10.egg/lithops/tests/test_storage.py", line 177, in test_storage_list_objects self.assertEqual(extract_keys(foo_objects), sorted([ AssertionError: Lists differ: ['__lithops.test/foo_baz', 'lithops.test/foo/baz', 'litho[46 chars]baz'] != ['lithops.test/foo/bar/baz', 'lithops.test/foo/baz', '__l[46 chars]baz']


Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
JosepSampe commented 1 year ago

Thanks @spiogit

JosepSampe commented 1 year ago

Storage test fixed in #1016