Rewrote the logic that populates bundle metadata and prepares bundles for download.
Related Issue(s)
Closes #201
Changes Made
List the main changes made in this PR. Be as specific as possible.
[ ] Feature added
[ ] Bug fixed
[x] Code refactored
[ ] Documentation updated
[ ] Other changes: [describe]
Checklist
Before submitting this PR, please make sure that:
[x] Your code passes linting and coding style checks.
[x] Documentation has been updated to reflect the changes.
[x] You have reviewed your own code and resolved any merge conflicts.
[x] You have requested a review from at least one team member.
[x] Any relevant issue(s) have been linked to this PR.
Additional Information
The old logic downloaded bundles from the SDA to populate it's metadata. The new logic populates bundle metadata by reading it from the dataset.bundle_size column, and downloading bundles from SDA is no longer needed. This way the bundles can be prepared for download much faster.
After the bundles are prepared for download in the CPA release, I will be doing a second release immediately after that removes the bundle_size column.
Description
Rewrote the logic that populates bundle metadata and prepares bundles for download.
Related Issue(s)
Closes #201
Changes Made
List the main changes made in this PR. Be as specific as possible.
Checklist
Before submitting this PR, please make sure that:
Additional Information
The old logic downloaded bundles from the SDA to populate it's metadata. The new logic populates bundle metadata by reading it from the
dataset.bundle_sizecolumn, and downloading bundles from SDA is no longer needed. This way the bundles can be prepared for download much faster.After the bundles are prepared for download in the CPA release, I will be doing a second release immediately after that removes the
bundle_sizecolumn.