search

opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
68 stars 15 forks source link

cloudsql index creation on the pipeline #1602

Closed ravenac95 closed 3 months ago

ravenac95 commented 3 months ago

Which area(s) are affected? (leave empty if unsure)

No response

To Reproduce

See the logs here: https://github.com/opensource-observer/oso/actions/runs/9394143568

Describe the Bug

If you dig into the logs it seems to be attempting to create an index that already exists. So some idempotent logic isn't idempotent, a deletion isn't happening when it should, or there is some kind of naming clash.

Expected Behavior

Shouldn't fail

ryscheng commented 3 months ago

Oh ya, Carl and I discussed this today. There's this really weird thing happening where after you create an index and it successfully returns, the index is still missing. What's happening in this is that we keep trying to create the index, but it's still missing after every retry, and then after max_retries it raises an empty exception.

In this PR today I just have it give up after 3 attempts https://github.com/opensource-observer/oso/pull/1598

This doesn't mean we have all indices, it just gives up retrying without raising an exception. TBH I'm not sure what's going on with Postgres, but I'm ready to give up on this for now. I'm hoping a better OLAP database won't have this issue?

ryscheng commented 3 months ago

We should just move to an OLAP DB