SnowEx / snowexsql

A project to access the SnowEx database holding data from the NASA SnowEx campaign
https://snowexsql.readthedocs.io/en/latest/readme.html
GNU General Public License v3.0
25 stars 6 forks source link

Campaign observations - Abstract Point and Image data metadata #156

Closed jomey closed 1 month ago

jomey commented 1 month ago

This is quite a big one and refactors the structure around metadata for Point and Image data as discussed in #137 and closes related issues (bottom of this description)

It also updates the test suite to use factory boy along with the integration into ptytest.

I will highlight more major changes in code comments.

Grab yourself a :coffee: (or :beer:) this PR has a lot to unfold.

Close #137 Close #136 Close #132

jomey commented 1 month ago

I am out of ideas why tests for Python 3.9 through 3.12 are hanging with GitHub actions. Maybe @micah-prime or @aaarendt could check out this PR and run it locally. Of course, all tests run fine with my local setup and Python 3.12

jomey commented 1 month ago

I am out of ideas why tests for Python 3.9 through 3.12 are hanging with GitHub actions. Maybe @micah-prime or @aaarendt could check out this PR and run it locally. Of course, all tests run fine with my local setup and Python 3.12

Well now also the tests for 3.8 are hanging and I only added a test case that checks for more information in the connection string.

When I shutdown my DB locally, the tests start failing fairly quickly. Makes me think it is not a DB connection issue

aaarendt commented 1 month ago

The tests hang for me when it gets to test_db.py. The api and tables tests succeed.

I'm looking at the content of pg_stat_activity while the test hangs and see this:

image

Wondering if it's something related to postgresql being strict about locking concurrent transactions, and your new use of rolling back after each test? Maybe useful: https://github.com/sqlalchemy/sqlalchemy/issues/4685

jomey commented 1 month ago

The tests hang for me when it gets to test_db.py. The api and tables tests succeed.

I'm looking at the content of pg_stat_activity while the test hangs and see this:

Wondering if it's something related to postgresql being strict about locking concurrent transactions, and your new use of rolling back after each test? Maybe useful: sqlalchemy/sqlalchemy#4685

Thanks for the great detective work @aaarendt I have updated the test suite and removed the rollback with the old session setup. I am trying not to overhaul the whole test suite with this PR and do this in a separate effort. Might have to bite the bullet though and at least replace the tests in test_db_tables with the new structure.

aaarendt commented 1 month ago

For my own learning, is it correct that e.g. lines 37-39 in api/test_layer_measurements.py (with reference to "fakeinstrument") is an example of old tests that will be superseded by your factory boy approach? (weird that you can't comment on code that hasn't been changed in a commit...?)

jomey commented 1 month ago

For my own learning, is it correct that e.g. lines 37-39 in api/test_layer_measurements.py (with reference to "fakeinstrument") is an example of old tests that will be superseded by your factory boy approach? (weird that you can't comment on code that hasn't been changed in a commit...?)

Correct. Just logged an issue for that #158