search

responsible-ai-collaborative / aiid

The AI Incident Database seeks to identify, define, and catalog artificial intelligence incidents.
https://incidentdatabase.ai
Other
171 stars 35 forks source link

Realm staging timing out & tests failing #837

Closed cesarvarela closed 1 year ago

cesarvarela commented 2 years ago

Something happened on staging Realm. It is timing out on some queries that work ok on production, and I can't reproduce it on my Realm dev instance with the same dataset and same Realm app.

works:

https://incidentdatabase.ai/api/graphql?query=query%20%7B%0A%20%20reports(limit%3A%209999)%20%7B%0A%20%20%20%20report_number%0A%20%20%7D%0A%7D

time outs:

https://staging-aiid.netlify.app/api/graphql?query=query%20%7B%0A%20%20reports(limit%3A%209999)%20%7B%0A%20%20%20%20report_number%0A%20%20%7D%0A%7D%0A

@smcgregor is staging on a lower tier? Maybe upgrading that helps but before that can you try mongorestoring staging from production?

smcgregor commented 2 years ago

Staging is indeed on a slightly lower tier, but they are both on pretty low tiers. I'm on the move at the moment, but I can clear staging for you to restore it from a production snapshot. Give me a message when I should clear the staging collections from the web UI and you can restore it immediately after. Does that work?

On Thu, Jul 21, 2022, 4:09 PM Cesar Varela @.***> wrote:

Something happened on staging Realm. It is timing out on some queries that work ok on production, and I can't reproduce it on my Realm dev instance with the same dataset and same Realm app.

works:

https://incidentdatabase.ai/api/graphql?query=query%20%7B%0A%20%20reports(limit%3A%209999)%20%7B%0A%20%20%20%20report_number%0A%20%20%7D%0A%7D

time outs:

https://staging-aiid.netlify.app/api/graphql?query=query%20%7B%0A%20%20reports(limit%3A%209999)%20%7B%0A%20%20%20%20report_number%0A%20%20%7D%0A%7D%0A

@smcgregor https://github.com/smcgregor is staging on a lower tier? Maybe upgrading that helps but before that can you try mongorestoring staging from production?

— Reply to this email directly, view it on GitHub https://github.com/responsible-ai-collaborative/aiid/issues/837, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAP2DE26W3AL5OZZKND2VLVVHKBFANCNFSM54JJIMNA . You are receiving this because you were mentioned.Message ID: @.***>

cesarvarela commented 2 years ago

I actually was able to do that from the console, but still, it didn't 'work, so I created another env and pointed staging Netlify to it:

https://staging-aiid.netlify.app/

(if you want to play around with translations stuff, you have to run on the console localStorage.setItem('i18n', 1))

So there is something else going on with staging Realm (or now called Atlas) because on this new env tests and build run faster and consistently 🤔

smcgregor commented 2 years ago

You created a new MongoDB staging environment? Are you able to add me to it?

Should I do a complete user-centered testing of staging for Monday's deployment to production?

cesarvarela commented 2 years ago

You created a new MongoDB staging environment? Are you able to add me to it?

Invite sent, I also created an app user using the same credentials of the old staging.

I think we should go back to the old staging anyway, but there is something going on that the same graphQL query may take 5 seconds or 50 seconds, and I didn't want to be blocked by that.

Should I do a complete user-centered testing of staging for Monday's deployment to production?

yes please

smcgregor commented 1 year ago

Closing this since I think it was addressed. Please reopen if I am mistaken.