Feature/sckan 275

[afonsobspinto]

Closes https://metacell.atlassian.net/browse/SCKAN-275

• Updates the django models to support region + layer as discussed here

• Adds migration to move from the old model to a new model without losing data. The migrations are split into 4 steps:

  1. Add the new models and new properties to the models that are going to be affected (Connectivity Statement, VIas and Destinations)
  2. Copy the old properties of the models affected to the new properties created in 1 (f.e copy ConnectivityStatement.origins to ConnectivityStatement.origins_new)
  3. Remove old AnatomicalEntity model and properties that referred to it
  4. Rename new properties to the names used previously

• Updates django admin page to add basic support for the new region + layer model (might be enough to close https://metacell.atlassian.net/browse/SCKAN-279 but I didn't put any thought into it, I just added what I need for testing purposes cc @D-GopalKrishna )

• Updates serializers to be compliant with the new model just enough for the application doesn't crash ( it's possible that this will change again in the scope of https://metacell.atlassian.net/browse/SCKAN-276)

• Updates command to ingest anatomical entities

• Updates command to ingest statements from neurondm. It includes a small refactoring that splits different responsibilities of the script per different files. The neurondm script was changed to keep the region + layer information. The cs ingestion service was changed to be compliant with the new model and also to allow the 'moving' a generic anatomical entity (meta) into a specific (region or layer) anatomical entity (meta) when the flag update_anatomical_entities is set. Updates changes detector algorithm and validators to work with complex entities.

• Generates new openapi specification and frontend stubs

• Modifies the frontend just enough for the application not crash on the most basic functionalities (requires further testing cc @ddelpiano )

[afonsobspinto]

I think it might make more sense to have anatomical entities as an abstract model and have as children:

• Simple Anatomical Entity
• Layer
• RegionLayer

[afonsobspinto]

I updated the description of the PR accordingly to the most recently changes.

For tracking purposes, this is what the original description was:

- Updates composer model to have both region and layer as anatomical entities - Update neuronDM script to retain region + layer information - Update ingest statements command to create region and layers when needed.   

[afonsobspinto]

@zsinnema I did not had the time to try any of the speed improvement suggestions on the admin page yet. Would it be possible to merge this PR only with the usability changes so that @ddelpiano can test it?

[zsinnema]

@afonsobspinto imo this is not mergable, see the attached video. In this video I open a (one) ConnectivityStatement in the admin using this link https://127.0.0.1:8000/admin/composer/connectivitystatement/296/change/ (I ingested the dev database)

https://github.com/MetaCell/sckan-composer/assets/124044/843f6f1f-0933-4856-a34e-9e519e5317a3

[afonsobspinto]

@zsinnema Do you believe this is not mergeable do to the high amount of queries? If that's so, I agree with you that it's something that could be improved but my point is that it's not something that is responsibility of this PR. Have a look at the SQL query count in both dev and local with dev database:

Dev:

Local with dev database:

[zsinnema]

@afonsobspinto herewith a video with the dev db on my local machine and git branch develop. as you can see the amount of queries is a "million" times less

https://github.com/MetaCell/sckan-composer/assets/124044/53d69ff1-f3aa-4afc-956f-05ec6e00d488

[zsinnema]

@ddelpiano The performance decreased is caused by the chanegs done in this PR/feature. Therefore I classify the implementation as being not mergable. I understand that the client also needs to see things/progress so the question is: how big is the is the performance decrease for the usability of the application. In the past we faced a perf decrease and then it was a big thing. Of course we could say that the performance decrease is a "bug". But the thing is that we already know the bug and the PR isn't merged yet.

Up to you if you think we can merge it or not

[afonsobspinto]

@zsinnema I can't replicate the behaviour of having such high amount of queries when running with this branch with the dev database:

https://github.com/MetaCell/sckan-composer/assets/19196034/d32a065d-898a-4aab-9355-3a08d2305360

This is what I did:

• docker-compose -f docker-compose-db.yaml up -d
• cat composer.sql | docker exec -i 30c8dea28dc9 psql -U <User> -d <Password>
• USE_PG=1;DEBUG=1 python manage.py migrate
• USE_PG=1;DEBUG=1 python manage.py ingest ingest_statements --update_upstream --update_anatomical_entities
• USE_PG=1;DEBUG=1 python manage.py ingest runsslserver

[zsinnema]

@afonsobspinto hmmm after I pulled again from Git the many queries disappeared... weird