search

PolyJIT / benchbuild

Empirical-Research Toolkit
MIT License
11 stars 12 forks source link

add source to download multiple files via HTTP #546

Closed jonas-kaufmann closed 1 year ago

jonas-kaufmann commented 1 year ago

Hi, for some experiments in https://github.com/se-sic/VaRA-Tool-Suite, we need to download multiple files over HTTP. This is currently a bit cumbersome as it requires us to specify multiple HTTP sources. @vulder had the idea to implement a source class that allows you to specify the url and then the files you want to download, like so:

class HTTPMultiple(
    local="geo-maps",
    remote={
        "1.0":
            "https://github.com/simonepri/geo-maps/releases/"
            "download/v0.6.0/"
    }
    files = ["countries-land-1km.geo.json", ...]
)

Let me know, what you think.

simbuerg commented 1 year ago

Hi,

sounds like a good idea to me. Thanks for working on this!

simbuerg commented 1 year ago

Had a further look. I should read my own code more... :-) This is fine, you return the cache path as a directory. HTTP will take care of the copy to build directory for you.

LGTM.

jonas-kaufmann commented 1 year ago

Had a further look. I should read my own code more... :-) This is fine, you return the cache path as a directory. HTTP will take care of the copy to build directory for you.

LGTM.

No problem but yes, that's what I intended :)