search

iipc / warc-specifications

Centralised repository for WARC usage specifications.
http://iipc.github.io/warc-specifications/
100 stars 30 forks source link

WARC revision 1.1 (clarification): validity of a WARC record with unknown field(s). #9

Closed cleymour closed 6 years ago

cleymour commented 9 years ago

Definition: In 6.1 “WARC record types/General”, it is written “Because new record types that extend the WARC format may be defined in future standards, WARC processing software shall skip records of unknown type”.

However, it is not explained if a known record type with new named fields should be valid; and should be processed by WARC reading or validating tools.

Decision: it should be stated that it is allowed to add new named fields, as well as new records types. However, it is strongly recommended to discuss the utility and the relevance of these new named fields within the right forum (e.g. the IIPC for web archiving) and to inform the maintenance body of the standard (currently the ISSN International Centre).

New named field that are considered relevant and mature by a large community of WARC users should be added in the next revision of the WARC standard.

Besides, the sentence “Because new record types that extend the WARC format may be defined in future standards…” should be replaced by “Because new record types that extend the WARC format may be defined in future versions of the standard…”.

Action: Clément Oury to propose a formulation.

cleymour commented 9 years ago

(1) Add in section 4 "File and record model", add to the list of defined fields | WARC-Future-type ; as defined by the concerned user community and the maintenance agency

cleymour commented 9 years ago

(2) Add at the end of section 4 (after "as specified in 8.3."): "New records types and new named fields may be added in addition to the existing list. However, it is strongly recommended to discuss any addition before implementing it. Discussion should be held within the right forum (e.g. IIPC for web archiving) under the auspices of the maintenance agency of the standard. New records types and new named fields agreed by the community of WARC standard users in the relevant domain should be mentioned in the further version of the WARC standard".

cleymour commented 9 years ago

(3) Rephrase the second sentence of 6.1 "General": "Because new record types that extend the WARC format may be defined in future versions of the WARC standard, WARC processing software shall skip records of unknown type".

anjackson commented 9 years ago

Wording should make it clear that this is a guideline to avoid collisions, check a suitable field doesn't already exists, and to facilitate future adoption/standardisation.

saraaubry commented 9 years ago

The following changes have been integrated in the revised ISO draft during the ISO working group meeting on November 16-17, 2015 :

1/ Add in section 4 "File and record model": New named fields and new records types may be defined in extensions of the core format. However, it is strongly recommended to discuss any addition to verify that a suitable field or type doesn't already exist to avoid collision. Discussion should be held within the right forum (e.g. the IIPC for web archiving) and the maintenance agency of the standard should be informed of any addition to facilitate the future revision of the standard.

**Check with ISO how to define maintenance agency.

2/ Delete in section 5.5 "Warc-Type" in the example " | future-type future-type=token".

3/ Rephrase the second sentence of 6.1 "General": "Because new record types that extend the WARC format may be defined in future versions of the WARC standard, WARC processing software shall skip records of unknown type".

4/ Delete in section 5.13 "Warc-Truncated" in the example " | future-reason future-reason=token" and replace it by the sentence : "Other reasons may be defined in extensions of the core format."

saraaubry commented 6 years ago

Included in WARC 1.1