Refactor Content Updates

gyorilab / indra_db

A Database-based knowledge back-end built on and for INDRA. The INDRA Database is a service that can be set up by any user with their own content and knowledge access. Our implementation of the database is the back-end to many of our projects, providing a vast and detailed knowledge base derived from many resources.

GNU General Public License v3.0

16 stars 10 forks source link

Refactor Content Updates #171

Closed pagreene closed 3 years ago

pagreene commented 3 years ago

This PR significantly refactors how content updates are performed, removing multiprocessing and focusing on implementing a more usable API. The PR also continues the trend of putting more database logical work into SQL that is executed on the remote server. Specifically, the process of getting the new primary key IDs from the ref table after new refs are added.

This PR also adds the pub_year column to text refs, and license column to the text content.

pagreene commented 3 years ago

There are some notable optimizations that could be made, but which are out-of-scope for this particular PR. Specifically:

The retrieval of IDs from the database, although generally an improvement over what came before, is still slower than I think it could be. I believe the joins used are not very well optimized.
Extracting individual files from an archive is very slow, and only pays off if you are only getting a few of the files from the archive. Otherwise it is faster to extract the entire archive and then grab the files you want. At this point, it is generally known what fraction of an archive is going to be extracted beforehand, so the optimization calculation could be made fairly easily.

pagreene commented 3 years ago

It has also been a long time since a deep review of how we resolve conflicting refs, and that entire process could use a deep dive, which was also well outside the scope of this PR.