search

metanorma / pubid-nist

BSD 2-Clause "Simplified" License
1 stars 2 forks source link

Progress bar for CLI report command or another solution to speedup NIST Tech Pubs xml download #123

Open mico opened 2 years ago

mico commented 2 years ago

I just found https://raw.githubusercontent.com/usnistgov/NIST-Tech-Pubs/nist-pages/xml/allrecords.xml 52Mb It takes around 15 seconds to complete download on 50Mbps internet. But with slower internet could take minutes, so the user could think "report" command is just broken. Tried to find compressed version to download but didn't find any.

mico commented 2 years ago

Also, maybe it's a good idea to save a file to the cache e.g. ~/.cache/nist-pubid/nist-tech-pubs.xml and update it (once per day?) after report displayed

Could use something like this https://github.com/DannyBen/webcache ?

mico commented 2 years ago

@ronaldtse For now I just added the message "Downloading NIST-Tech-Pubs XML database...". Will it be enough for now? Should I work on caching?

ronaldtse commented 2 years ago

@mico the "down" gem is actually pretty good at showing progress: https://github.com/janko/down

Yes I think we need to provide caching. Maybe you can use the path ~/.metanorma/cache/nist-pubid/nist-tech-pubs.xml.

I don't know how to integrate webcache functionality with down though, it might be possible.

In addition, I think adding some more messages around the download task is useful: