openlawlibrary / taf

The Archive Framework
GNU Affero General Public License v3.0
10 stars 10 forks source link

Fully support partial update #559

Open renatav opened 1 month ago

renatav commented 1 month ago

Description (e.g. "Related to ...", etc.)

Fully supporting partial update means adding support for excluding certain target repositories and storing information about what was validated and up to which auth commit. last_validated_commit is only updated if clone and update are run without excluding any target repositories. We have no way of knowing if the whole set of repos is valid unless all target repos are validated too. This PR introduces a new file, called last_validated_data, where we store last validated commit for each repository. These repositories will only be updated from those commits. The reason why I didn't opt for storing everything in this file is backwards compatibility. This all makes the update process a bit more complicated because different repos are validated from different auth commits.

Closes #251

Code review checklist (for code reviewer to complete)

renatav commented 2 weeks ago

@n-dusan If I remember correctly, there were a couple of components that were reading from this file. You're probably more familiar with the changes that were made in the last two years, so that might not be the case. I'll let you make this call

n-dusan commented 2 weeks ago

@renatav Thanks for the context! I’m not entirely sure if anything still reads from this file either. Happy to chat over it if you think that’d be helpful. Otherwise, I’ll dig a bit more and keep you posted.

renatav commented 2 weeks ago

@n-dusan Whatever you think would be easier. I just don't want there to be a fire down the line because LVC's structure was updated, leading to stressful debugging sessions.