nix-community / nixbox

NixOS Vagrant boxes [maintainer=@ifurther]
https://app.vagrantup.com/nixbox/
MIT License
304 stars 101 forks source link

iso_urls_update.rb web scrape broke #59

Closed shajra-simspace closed 4 years ago

shajra-simspace commented 4 years ago

I think the web site put in a newline that breaks the assumption of the regex parse. I haven't had a moment to sort out the way I'd prefer to fix it, but I thought I'd issue the bug here now. I may put in a PR to fix it when I get the time.

In the meantime, it's easy enough to edit the generated JSON file manually.

shajra-simspace commented 4 years ago

I'm not on the right computer to set up a PR, but I had a moment to write a version of iso_usls_update.rb that seems to fix the break. I switched from regex parsing to XML parsing, which hopefully is more robust.

#!/usr/bin/env nix-shell
#!nix-shell -i ruby
#
# Heuristic to update the ISO urls
#

require 'open-uri'
require 'json'
require 'rexml/document'
include REXML

isos = {}

xml = Document.new(open("https://nixos.org/nixos/download.html"))
xml.root.elements.each("body/*/*/*/ul/li") { |li|
  hrefs = li.get_elements("a").map { |a| a.attribute("href") }
  if hrefs.size == 2 and /.*minimal.*/i =~ hrefs[0].value
  then
    iso_url = hrefs[0].value
    arch_re = /https:\/\/.+-([^-]+)-linux.iso/
    arch = arch_re.match(iso_url).captures
    iso_sha256 = open(hrefs[1].value).read.strip.split.first
    if arch.size == 1
    then
      isos[arch[0]] = {
        iso_url: iso_url,
        iso_sha256: iso_sha256
      }
    end
  end
}

out = JSON.pretty_generate(isos)
puts out
File.write("iso_urls.json", out)
cdituri commented 4 years ago

Thanks for posting your fix @shajra-simspace, hit the same issue while working to finish #58 cdituri/feature/hyperv-packer-build and confirmed you're solution is working 👍

I'm not on the right computer to set up a PR

Caught this comment, and took your code into my own branch to finish the hyperv builder; made sure to attribute you in commit https://github.com/nix-community/nixbox/commit/1201013e103228f80683efd5e224fff78b7e8a22. If you're ok with this approach, can we continue discussion here https://github.com/nix-community/nixbox/pull/58#issuecomment-619654630 ?

zimbatm commented 4 years ago

merged as part of #58. Thanks!