Open tbreloff opened 8 years ago
I definitely don't have all the answers, but I have some ideas and some thought on how it works for me/how I plan for it to work with ControlSystems.jl:
git clone
could then suffice, without the need of using the package manager.Using git to download a binary resource is overkill and not really necessary. You probably don't need version control at the download site, only at the repository level. Just download an archive of a tagged version, then you can change the tag as needed. Please don't clone master of things, that's really bad for reproducibility and being able to go back to old versions and test them down the road.
You probably don't need version control at the download site, only at the repository level
I don't understand what you mean by this. When you say "download site", do you mean the computer which is running a test? When you say "repository level", what do you mean?
Please don't clone master of things, that's really bad for reproducibility and being able to go back to old versions and test them down the road.
It's no worse than using an image server with a snapshot of the reference images (which is what I think you're suggesting)
Maybe you could be a little more specific with what you're suggesting, and give an example workflow? Where are the images stored? How/when/where do I upload/download? How do we connect a package version/commit with the correct reference image for reproducibility? If the images are served statically, who hosts them?
You could use a github repo to hold the images if you want, it's not ideal but it's easy. But where the tests get run, I would be surprised if you needed the full history of all past versions of the data. For downloading the data, rather than a git clone I'm suggesting downloading and extracting the archive that github can create for you from any tag or sha. The test script would just contain the particular tag or sha to test against, download and extract that.
Testing visual output is a complex art... much trickier than most testing, mainly because of the fuzzy nature of what constitutes a "pass". I want to discuss best practices for visual testing in Julia... not what people do now, but rather what we should do. There are a few issues which we need good solutions for:
ref: https://github.com/JuliaLang/METADATA.jl/pull/5545 https://github.com/JuliaLang/METADATA.jl/pull/5494
cc: @tkelman @mfalt