Closed Mr0grog closed 3 years ago
Actually, the more I think about it, the less I want to propose adding charset
.
In a discussion yesterday, @danielballan concurred with the ideas here, so I feel a little better about actually doing it.
The removal of media_type_parameters
is already under way with #778.
Things that still need doing here:
capture_url
→ url
uri
→ body_url
version_hash
→ body_hash
content_length
→ body_length
headers
(a.k.a. source_metadata.headers
→ headers
)
The
Version
model has had a tortured history, starting as a local copy of Versionista’s data, and slowly evolving into something much different. Today it could probably use some cleanup.Normally I’d say this is not worth the bugs it might introduce, but we developer involvement here is waning, and I think it’s valuable to do the work to make the data easier to learn and understand, whether that’s to make future development by other programmers easier to get started with or to make dead, archived data easier to use.
Current fields:
✅ = all good as-is, ⚠️ = should change, :x: = should remove, 🌱 = should add
uuid
(capture_time, capture_url, source_type)
created_at
updated_at
capture_time
capture_url
https://epa.gov/climate-change/
. I think we should rename this tourl
, which would be more concise and clearer. Component of the natural key.source_type
internet_archive
. Component of the natural key.status
uri
body_url
for clarity. Everyone new gets tripped up on this one.version_hash
body_hash
for clarity, and to go with the suggested rename ofuri
tobody_url
above.source_metadata
Version
.page_uuid
title
content_length
Content-Length
header). Doesn’t reside anywhere else in the DB, but is/can be automatically derived from the data found from theuri
field. I’d like to suggest renaming this tobody_length
orbody_size
for consistency with the otherbody_*
fields suggested above and to differentiate from the header of the same name, but don’t think it’s a big deal. The current name is not a problem.media_type
Content-Type
header. I don’t think the name should change, although we should consider sniffing content and canonicalizing the type here instead of just reflecting the header. (See also #752)media_type_parameters
different
headers
Version
as a canonical record of an archived HTTP response, which is what it should really be at this point. We do not always have headers (so this must be nullable, or maybe an empty object?), but when we do they should be here instead of insource_metadata
.charset
~uri
/body_url
and, for HTML, it is often specified (and specified more correctly) in the document itself. It can also be retrieved from the headers in cases where we didn’t have to sniff it during import.~We also track redirects in
source_metadata
(so we know if the response is a direct response tocapture_url
or a response to a redirect from it), but I’m not sure that’s worth pulling up into the main model.I’ve made a point of calling out DB-centric vs. derived vs. denormalized because I think these are particularly useful distinctions. If there weren’t practical concerns, I’d almost suggest the derived & denormalized data belongs in a second table that has a 1:1 relationship with versions. That way we are separating out the canonical record of an archived HTTP response from other useful data we want to make easily available about it.
If it weren’t a total mess to do, I’d also suggest renaming
Version
toCapture
orSnapshot
, but I don’t think that’s worth the complex mess and migration headaches. Renaming fields is small potatoes in comparison.