Release iblapp in production

int-brain-lab / iblalyx

MIT License

0 stars 0 forks source link

Release iblapp in production #51

Closed oliche closed 3 months ago

oliche commented 6 months ago

Although I did release the app before the holidays, Mayo experienced problems linked to django-filters.

It turned out to be a Django 5.0 incompatibility issue. It would be nice to run the tests as part of deploying a newly built container, and test a new container with version 5.0 once this is resolved https://github.com/cortex-lab/alyx/issues/828

oliche commented 5 months ago

[x] upgrade to django 5 codebase
[x] add ibl_reports pages
[x] set cache with s3
[x] set paired recording cache with s3 and pqt file building instructions
[ ] look into moving domains to route 53
[ ] release openalyx clone
[x] check SSL certificates expirations

oliche commented 5 months ago

The server drops the connection when many small queries are made in a short amount of time:

from one.api import ONE
from brainwidemap import bwm_query # womp womp

one = ONE(base_url="https://cjqss5isu6.eu-west-2.awsapprunner.com", mode='remote')
bwm_df = bwm_query(freeze='2023_12_bwm_release', one=one, return_details=True)

for i, rec in bwm_df.iterrows():
    print(i, rec.eid)
    ses = one.alyx.rest('sessions', 'list', id=rec.eid, no_cache=True)

After a few iterations, the requests fail wit the following message: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

oliche commented 5 months ago

After further investigation, using a local instance with the exact same server parameters doesn't drop connections. So this doesn't seemed linked to gunicorn and most likely due to the AWS service way of handling requests.

k1o0 commented 5 months ago

When you receive a 104 response is there anything in the service logs? It's possible there's an AWS service quota limit reached, which could be between EC2, RDS or S3. So far I haven't been able to reproduce the error with the above code...

oliche commented 3 months ago

The app is potentially a good solution, but it is not mature enough for our purposes as there is no SSH access, and it is very hard to understand where the issue arises. We are reverting to a conventional docker deployment on an EC2 instance.