andrew2net
commented
1 year ago @ronaldtse I've tried all the codes from 00 to 99
responds = (0..99).map do |n|
resp = Faraday.get "https://www.iso.org/contents/data/ics/#{n.to_s.rjust(2,'0')}.rss"
n if resp.status == 200
end
responds.compact
=> [1, 3, 7, 11, 13, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 43, 45, 47, 49, 53, 55, 59, 61, 65, 67, 71, 73, 75, 77, 79, 81, 83, 85, 87, 91, 93, 95, 97]
responds.compact.size
=> 40How do you think, can we use these ICS codes only to obtain the full list of published standards? Or the RSS's responses can vary from time to time, and we need to try each of the 100 codes every time?
ronaldtse
Due to the slow retrieval from ISO's site we have to build a static dataset.
The method to build the static dataset is through ISO's RSS feeds.
For each ICS code, they have an RSS feed:
An ICS RSS feed provides a list of standards like this:
Steps to retrieve standards
This way we can enumerate all published standards and the latest stages/dates.
Detecting updates
From the daily retrieval of ICS RSS feeds, we can detect if there are any changes to the documents as the RSS feeds provide the latest publication/stage dates. For items that have been updated, we can update using their individual page links.