[Cloud Support] Setup development environment

[itutu-tienday]

Tasks

• [x] Setup development environment
• [x] Confirm application specs
• [x] Test cases

Documents

• Environment Setup Procedure
  • https://drive.google.com/file/d/1tJVsgJIaFn-oHSeTgKLSETIaoFmt-BBf/view?usp=drive_link
• Spec Documents
  • https://docs.google.com/presentation/d/1slp8oZSuriSWArkrQKrMnY9SLHw4bOgL/edit?usp=drive_link&ouid=104622411264052056363&rtpof=true&sd=true
  • https://drive.google.com/drive/folders/1l02DPAEBgSkUA9aMggm1ZcwA-oNC82W3
• Testcase
  • https://drive.google.com/drive/folders/1CSBbJlL4vIaWJDoS5IyCWjz6Rv0a-bKO

[milesAraya]

Created a S3 bucket and linked it to optinist. Running optinist transferred the files. correctly

[milesAraya]

Required steps

1. Create IAM user with CLI access (access key and secret key)
2. Add an S3 bucket
3. Add access key to aws configure
4. Setup multi-user mode optinist. Add to .env file:
   1. REMOTE_STORAGE_TYPE="2"
   2. OFFLINE_STORAGE_DIR="/tmp/optinist/mock-storage"
   3. S3_STORAGE_URL="s3://{MYS3NAME}"
   4. AWS_ACCESS_KEY_ID
   5. AWS_SECRET_ACCESS_KEY

steps 4.4 and 4.5 not in current walkthrough

[milesAraya]

To check if the download and sync function was working with S3, i tried uploading some files directly to S3. I wanted to see if they would be shown in optinist gui. However, this caused an error logging in to optinist. The user sees a confusing error:

Email or password is wrong

However, this is not a password error. Switching to REMOTE_STORAGE_TYPE="0", or resetting the password does not fix this login error.

Checking the log, the error seems to be related to inability to download files correctly

RemoteStorageController().download_all_experiments_metas()

subprocess.CalledProcessError: Command 'aws s3 sync s3://optinist/output

studio-dev-be-1  | 2024-08-29 23:03:09,223 ERROR:    [uvicorn.error] run_asgi():413 - Exception in ASGI application
studio-dev-be-1  | Traceback (most recent call last):
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
studio-dev-be-1  |     result = await app(  # type: ignore[func-returns-value]
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
studio-dev-be-1  |     return await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 1106, in __call__
studio-dev-be-1  |     await super().__call__(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
studio-dev-be-1  |     await self.middleware_stack(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
studio-dev-be-1  |     raise exc
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
studio-dev-be-1  |     await self.app(scope, receive, _send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 91, in __call__
studio-dev-be-1  |     await self.simple_response(scope, receive, send, request_headers=headers)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 146, in simple_response
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
studio-dev-be-1  |     raise exc
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
studio-dev-be-1  |     await self.app(scope, receive, sender)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
studio-dev-be-1  |     raise e
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
studio-dev-be-1  |     await route.handle(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
studio-dev-be-1  |     await self.app(scope, receive, send)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
studio-dev-be-1  |     response = await func(request)
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 274, in app
studio-dev-be-1  |     raw_response = await run_endpoint_function(
studio-dev-be-1  |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
studio-dev-be-1  |     return await dependant.call(**values)
studio-dev-be-1  |   File "/app/studio/app/common/routers/auth.py", line 26, in login
studio-dev-be-1  |     raise e
studio-dev-be-1  |   File "/app/studio/app/common/routers/auth.py", line 23, in login
studio-dev-be-1  |     RemoteStorageController().download_all_experiments_metas()
studio-dev-be-1  |   File "/app/studio/app/common/core/storage/remote_storage_controller.py", line 181, in download_all_experiments_metas
studio-dev-be-1  |     return self.__controller.download_all_experiments_metas()
studio-dev-be-1  |   File "/app/studio/app/common/core/storage/s3_storage_controller.py", line 97, in download_all_experiments_metas
studio-dev-be-1  |     cmd_ret = subprocess.run(
studio-dev-be-1  |   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
studio-dev-be-1  |     raise CalledProcessError(retcode, process.args,
studio-dev-be-1  | subprocess.CalledProcessError: Command 'aws s3 sync s3://optinist/output /tmp/tmprzqcwgg0 --dryrun --exclude '*' --include '*/experiment.yaml' --include '*/workflow.yaml' ' returned non-zero exit status 127.

[milesAraya]

It seems the issue was caused by inability to find AWS Keys using aws_s3_sync_command.

Fixed using boto3 instead of aws cli. See here for implementation