Specification

[DavidBuchanan314]

Please specify the file format.

[OttoCoddo]

Sure. For the time being, checking the reference code and emailing me can be helpful.

[nlfiedler]

I took a stab at implementing this in Rust. It was a very interesting endeavor, but ultimately unsuccessful due to it being too slow. In the README I describe the basic file format: https://github.com/nlfiedler/pack-rs

[OttoCoddo]

Interesting work, @nlfiedler. Thank you for taking the time. Here are some notes out of respect for your work:

• You should use a transaction.
• Read the latest Content ID from the database, or use 0 if it is a new Pack file. This way, you prevent inserting zero content to get its ID. Use a queue or a list to keep what ItemContent belongs to what Content, just like in the Pascal version.
• Do not allocate memory for each compression, reuse it. And use that memory to bind to SQLite. Using blob writes makes it much slower. I do not know if your SQLite binding gives you that, but it certainly should.
• I strongly suggest keeping the structure (column names and constants) the same, so this program output can be read by Pascal version of Pack and the other way around. 
• There is no need to add a text field to detect the compression format, if there will be any other in the future. They all have a header that you can read to detect. Here is the frame header you can read to check if it is a Zstandard frame or not. Or look for ZSTD_isFrame().

On your notes:

• Zstandard is the best choice for all around work. I tried almost all, and there is not much reason to use other ones unless you are looking for a niche use case. I suggest focusing on only Zstandard for now.
• Pack can handle small archives (less than 20KB) perfectly fine. The minimum size would be 2KB for a Pack file.
• Streaming can be done just fine. You can read each ItemContent of each Item ordered by ItemPosition and write it to output. Every design has it cons, but Pack tried to minimise them, and these well known points are considered in the design.

If you want to continue discussing this subject or more clarification on the file format, you can email me, o at pack.ac.

[nlfiedler]

Thank you for the suggestion about re-using the byte buffer, that helped significantly.

[OttoCoddo]

Welcome.

[thousandsofthem]

@OttoCoddo

• Are there any place where table structure is defined/documented? I've seen it somewhere, but having it officially documented would be great. Documentation on pack.ac is barebones. Having something like this https://github.com/nlfiedler/pack-rs?tab=readme-ov-file#item would be fantastic. Maybe keep it in GitHub repo and embed into website somehow? Either link, iframe or fetching on build step

• As far as I've seen the format, everything makes sense. Well, keeping original magic byte would be very useful on practice, and defining sqlite application format instead. Which means integrating Pack becomes walk in the park, by using existing tooling for whatever language you need.

[OttoCoddo]

• No, but I have prepared a draft and will publish it soon and notify you here too.
• Using the SQLite application version feature is a good choice, but not for a format that can be used countless times. It brings too much unintentional heat to the SQLite team and limits feature version updates. For enthusiasts or developers, they can change the header or just use the --activate-other-options --transform-to-sqlite3 parameter to transform a Pack file to SQLite3, and use the --activate-other-options --transform-to-pack to go back.