Closed blackfyre closed 6 months ago
The recent changes involve incorporating compressed file handling into the system's migration logic. The migration scripts have been updated to read .zst
compressed files, decompress them using a new utility function, and then process the JSON data. This update shows an enhancement in the application's capability to handle and process compressed data efficiently.
Files | Change Summary |
---|---|
.../1695699036_artists_table.go |
Introduced handling of .zst compressed files, decompression of data, and processing of the decompressed JSON. |
.../1696479790_add_art_table.go |
Replaced existing artwork data processing with new logic for reading and decompressing .zst files, followed by JSON unmarshalling. |
utils/zst.go |
Added a new utility function Decompress for decompressing data using zstd . |
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
Addition of new Reference Files
New reference files for artists and artwork have been added to the project. These files are titled artists_with_bio_stage_2.json.zst
and artworks_stage_2.json.zst
, respectively, and stored under the assets/reference
directory.
Inclusion of New Dependencies
The project now includes an additional dependency named github.com/klauspost/compress v1.17.4
. This has been registered in both go.mod
and go.sum
files. This package will assist in the compression of data, enhancing the efficiency of data utilization.
Changes to Artists Table Migration File
The existing migrations/1695699036_artists_table.go
file has seen some modifications. It now includes import statements for "bytes"
and "blackfyre.ninja/wga/utils"
, which help in managing data. The init
function within this file has also been modified to read and decompress the newly added artist reference file.
Changes to Art Table Migration File
The migrations/1696479790_add_art_table.go
file was updated in a similar fashion to accommodate the artwork reference file. The import statements for "bytes"
and "blackfyre.ninja/wga/utils"
were added here as well, extending the utility function of the file. The init
function now reads and decompresses the reference/artworks_stage_2.json.zst
file.
Addition of Decompression Function in the utils/zst.go File
A new file named utils/zst.go
has been created. It houses a Decompress
function which is responsible for decompressing zstd (Zstandard compression algorithm) files, further enhancing the efficiency of handling large data files within the project.
Summary by CodeRabbit
.zst
compressed files.