Should other directories besides archive/, indexes/ be part of the spec?

[ikreymer]

Or should spec be kept small, and allow for extension as needed?

The initial focus is replay, but perhaps other use cases have other requirements that are shared and could use a standardized approach?

[ato]

Crawl artifacts is one such thing that I think would be very useful to standardize.

Obvious reasons for splitting the spec up:

• The spec has become too large to comfortably navigate.
• So that implementations can advertise they fully support certain logical groups of functionality.
• Process reasons. If there's some sort of formal standardization process that makes the core spec difficult to change or there are multiple working groups working on different aspects of it with different release cycles.
• A lot of very niche things get suggested which most of the community doesn't care about.

I don't think any of those need to be worried about just yet. So my suggestion would be to keep it simple and start with one spec. You can always split it up later when things start becoming unwieldy and it'll be more obvious at that time how it should be split up.

[ikreymer]

Yeah,I've been leaning towards optimizing specifically for the functional use case of data that is needed for replay, or perhaps a text index, so that anything that is added to the spec has an implementation that is using that data. I think that will help keep the spec more lean, and I agree that it should be kept in sync with things that are using it.

Eg. if there is a tool that would analyze crawl artifacts in a specific way using WACZ, then it would make sense to add them, but not until then..

[ikreymer]

For now, we have archives/, indexes/ and pages/, all used for replay. The adoption of frictionless data package spec also includes a datapackage.json.

Closing this for now, will open additional issues with any other specific considerations.