arayabrain / barebone-studio

pipeline tool
GNU General Public License v3.0
1 stars 3 forks source link

[Cloud Support] Setup development environment #422

Closed itutu-tienday closed 1 month ago

itutu-tienday commented 2 months ago

Tasks

Documents

milesAraya commented 2 months ago

Created a S3 bucket and linked it to optinist. Running optinist transferred the files. correctly

Screenshot 2024-08-29 at 12 28 26 Screenshot 2024-08-29 at 12 29 18
milesAraya commented 2 months ago

Required steps

  1. Create IAM user with CLI access (access key and secret key)
  2. Add an S3 bucket
  3. Add access key to aws configure
  4. Setup multi-user mode optinist. Add to .env file:
    1. REMOTE_STORAGE_TYPE="2"
    2. OFFLINE_STORAGE_DIR="/tmp/optinist/mock-storage"
    3. S3_STORAGE_URL="s3://{MYS3NAME}"
    4. AWS_ACCESS_KEY_ID
    5. AWS_SECRET_ACCESS_KEY

steps 4.4 and 4.5 not in current walkthrough

milesAraya commented 2 months ago

To check if the download and sync function was working with S3, i tried uploading some files directly to S3. I wanted to see if they would be shown in optinist gui. However, this caused an error logging in to optinist. The user sees a confusing error:

Email or password is wrong

However, this is not a password error. Switching to REMOTE_STORAGE_TYPE="0", or resetting the password does not fix this login error.

Checking the log, the error seems to be related to inability to download files correctly

RemoteStorageController().download_all_experiments_metas()

subprocess.CalledProcessError: Command 'aws s3 sync s3://optinist/output

studio-dev-be-1  | 2024-08-29 23:03:09,223 ERROR:    [uvicorn.error] run_asgi():413 - Exception in ASGI application
studio-dev-be-1  | Traceback (most recent call last):
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
studio-dev-be-1  |     result = await app(  # type: ignore[func-returns-value]
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
studio-dev-be-1  |     return await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 1106, in __call__
studio-dev-be-1  |     await super().__call__(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
studio-dev-be-1  |     await self.middleware_stack(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
studio-dev-be-1  |     raise exc
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
studio-dev-be-1  |     await self.app(scope, receive, _send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 91, in __call__
studio-dev-be-1  |     await self.simple_response(scope, receive, send, request_headers=headers)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 146, in simple_response
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
studio-dev-be-1  |     raise exc
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
studio-dev-be-1  |     await self.app(scope, receive, sender)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
studio-dev-be-1  |     raise e
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
studio-dev-be-1  |     await route.handle(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
studio-dev-be-1  |     response = await func(request)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 274, in app
studio-dev-be-1  |     raw_response = await run_endpoint_function(
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
studio-dev-be-1  |     return await dependant.call(**values)
studio-dev-be-1  |   File "/app/studio/app/common/routers/auth.py", line 26, in login
studio-dev-be-1  |     raise e
studio-dev-be-1  |   File "/app/studio/app/common/routers/auth.py", line 23, in login
studio-dev-be-1  |     RemoteStorageController().download_all_experiments_metas()
studio-dev-be-1  |   File "/app/studio/app/common/core/storage/remote_storage_controller.py", line 181, in download_all_experiments_metas
studio-dev-be-1  |     return self.__controller.download_all_experiments_metas()
studio-dev-be-1  |   File "/app/studio/app/common/core/storage/s3_storage_controller.py", line 97, in download_all_experiments_metas
studio-dev-be-1  |     cmd_ret = subprocess.run(
studio-dev-be-1  |   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
studio-dev-be-1  |     raise CalledProcessError(retcode, process.args,
studio-dev-be-1  | subprocess.CalledProcessError: Command 'aws s3 sync s3://optinist/output /tmp/tmprzqcwgg0 --dryrun --exclude '*' --include '*/experiment.yaml' --include '*/workflow.yaml' ' returned non-zero exit status 127.
milesAraya commented 2 months ago

It seems the issue was caused by inability to find AWS Keys using aws_s3_sync_command.

Fixed using boto3 instead of aws cli. See here for implementation