search

nansencenter / nansat

Scientist friendly Python toolbox for processing 2D satellite Earth observation data.
http://nansat.readthedocs.io
GNU General Public License v3.0
179 stars 66 forks source link

Introduce Continuous Integration through Travis CI #80

Closed letmaik closed 9 years ago

letmaik commented 10 years ago

Continuing from #72 this issue is about enabling Travis CI builds to automatically test nansat on each commit.

letmaik commented 10 years ago

I got the travis build running so far, but now the tests are failing: https://travis-ci.org/nansencenter/nansat/builds/30265584

/home/travis/build/nansencenter/nansat/nansat/tests/nansat_test_archive.py:108: UserWarning: Could not access ftp-site with test data - contact morten.stette@nersc.no to get the ftp-server at NERSC restarted

@asumak @mortenwh I'm not sure what that tells me. Is this specific ftp server really down?

letmaik commented 10 years ago

Also, the following status image can be included in the README.md so that it is displayed on the github project page:

Build Status

It is related to a specific branch, the above is for develop. You could also display two images for the master and develop branches.

mortenwh commented 10 years ago

Yes, it unfortunately has a tendency to go down quite often.... Maybe there is a better place to put the data?

  1. juli 2014 15:58 skrev "Maik Riechert" notifications@github.com følgende:

I got the travis build running so far, but now the tests are failing: https://travis-ci.org/nansencenter/nansat/builds/30265584

/home/travis/build/nansencenter/nansat/nansat/tests/nansat_test_archive.py:108: UserWarning: Could not access ftp-site with test data - contact morten.stette@nersc.no to get the ftp-server at NERSC restarted

@asumak https://github.com/asumak @mortenwh https://github.com/mortenwh I'm not sure what that tells me. Is this specific ftp server really down?

— Reply to this email directly or view it on GitHub https://github.com/nansencenter/nansat/issues/80#issuecomment-49433418.

letmaik commented 10 years ago

Well, is seems as if the test data is quite huge and question would be whether nansat can be tested equally with smaller datasets, maybe even dynamically created ones. Then storage would be less of a problem.

Am 18.07.2014 17:50, schrieb Morten W. Hansen:

Yes, it unfortunately has a tendency to go down quite often.... Maybe there is a better place to put the data?

  1. juli 2014 15:58 skrev "Maik Riechert" notifications@github.com følgende:

I got the travis build running so far, but now the tests are failing: https://travis-ci.org/nansencenter/nansat/builds/30265584

/home/travis/build/nansencenter/nansat/nansat/tests/nansat_test_archive.py:108: UserWarning: Could not access ftp-site with test data - contact morten.stette@nersc.no to get the ftp-server at NERSC restarted

@asumak https://github.com/asumak @mortenwh https://github.com/mortenwh I'm not sure what that tells me. Is this specific ftp server really down?

— Reply to this email directly or view it on GitHub https://github.com/nansencenter/nansat/issues/80#issuecomment-49433418.

— Reply to this email directly or view it on GitHub https://github.com/nansencenter/nansat/issues/80#issuecomment-49446699.

knutfrode commented 9 years ago

Yes, I agree. Tiny (stored within repository) or dynamically created datasets would be better. Automatic download of huge datasets (order ~100 MB) is not a good idea, even if the server would be stable.

E.g. in subfolder 'tutorial' there is a 0.1 MB Tiff file (gcps.tif) used for testing. I guess this should be enough to test functionality?

mortenwh commented 9 years ago

If we should test the full functionality of nansat including the mappers, I think we need to have examples of all the datasets. Is it possible to make e.g. empty N1-files for testing the envisat mappers? How could we go forward with this?

letmaik commented 9 years ago

Maybe downsampling the datasets or extracting a small region (e.g. 50x50 pixels) is an option, but I don't know if that's easily possible.

knutfrode commented 9 years ago

Is it really necessary/meaningful to have tests for all mappers? I guess most important should be to test main Nansat functionality on (small samples of) different types of datasets, such as e.g. projected images (with geotransform), unprojected images (with GCPs), and perhaps also an image with Geolocation Grid.

For most users, several of the mappers will never be used. And users may have GDAL compiled without e.g. HDF4, HDF5 or netCDF-support, if they are not planning/needing to work with some of those formats. Some formats/drivers such as MSG also need specific compilation of GDAL. Then tests will give a lot of errors which are irrelevant.

Downsampling/subsetting while maintaining the exact file format is difficult/impossible for many file formats (including e.g. ASAR and Radarsat).

mortenwh commented 9 years ago

I get emails about errored builds in Travis CI like https://travis-ci.org/nansencenter/nansat/builds/33486069 when I push code - is it something to worry about? @akorosov does not get them...

mortenwh commented 9 years ago

@knutfrode - this is probably one possible solution but then we have to rely on people properly testing their mappers themselves, which could be a problem especially when existing mappers are modified. Another option could be to only use "small" datasets for testing but that will still be quite huge for some datasets (e.g. Sentinel-1 is on the order of GB's)...

In my opinion there should be a complete test suite which includes the mappers but perhaps it could be a separate package not in nansat? Then we could have the simple tutorial-like tests in the nansat package like you suggest.

knutfrode commented 9 years ago

Yes, I think a separate package would be better and cleaner. This is how it is done in GDAL, the "autotest"-suite is separated from the main code: http://trac.osgeo.org/gdal/browser/trunk Seems like most drivers are tested, but not all. And the test data are downloaded from a separate URL: http://trac.osgeo.org/gdal/browser/trunk/autotest/README

mortenwh commented 9 years ago

Great - so two new issues: 1: Move most of the current nansat tests to a separate package, e.g. nansat_tests 2: Make tests package (nansat.tests) based on the tutorial (should the tutorial be replaced by tests, or do we need both?)

Ok?

letmaik commented 9 years ago

Sounds good, just keep in mind that the point of having Travis CI is to test nansat on every commit to see if something breaks. I guess the separate tests package (separate github project probably) would then be used for manual testing before doing any new releases. So the tests in nansat itself should probably cover most of nansat's functions while leaving the mappers-specific ones for the separate test project.

@mortenwh I'll look into the build error. Seems like travis ci is using an unstable Ubuntu PPA repository which is currently broken. Let me try something.. EDIT: The installation is working now, I applied a temporary fix from travis-ci/travis-ci#2683 But now the actual tests fail, looks a bit like out of memory to me: https://travis-ci.org/nansencenter/nansat/builds/33498486#L779

mortenwh commented 9 years ago

On a second thought - as the installation now works fine, why not keep the tests as they are? The test data is actually downloaded, so access to the ftp-server should only be a risk the first time a test on a new dataset is attempted. Unless time and storage capacity is a problem, why not keep it as it is?

Note that tests on proprietary data that is not possible to share must have skip decorator to make sure the test is not run unless there is data.

@neothemachine Are the Radarsat-2 and Cosmo-Skymed files on Travis CI accessible from outside? If so, we should remove them.

letmaik commented 9 years ago

@mortenwh It is a question of resources. Do you want to have this amount of traffic for your ftp server on every commit? Also, the downloading takes time and leads to long tests, at the moment nearly half an hour which I find too long. Also, by default, Travis CI builds will timeout if there is no console output for more than 10 minutes. And given that it's a free service I wouldn't push too hard.

Regarding your question about Radarsat-2 and Cosmo-Skymed files, I don't really understand that. If Travis-CI can download something from an FTP, then of course everyone else can do the same. To still answer your question, no, it is not possible to download any files from the Travis CI machines that were produced during a test run, in fact, the virtual machine is destroyed/reset after each test run.

mortenwh commented 9 years ago

@neothemachine ok - that sounds good regarding the files, they are removed from the ftp now. Regarding the traffic for the ftp-server, I thought the files would stay on Travis-CI after the first download, but what you say then means it downloads data every time? That and the long tests then favours the suggested separation into one independent test package for mappers, and one like the tutorial within the nansat package.

letmaik commented 9 years ago

Yes, each test run is running on a fresh VM, totally independent. There is a caching feature but this is currently only available for the paid service.

letmaik commented 9 years ago

Note to self: Replace "sudo apt-get" with conda in travis ci script to test nansat against Python 2 and 3. There's a small tutorial here: https://gist.github.com/dan-blanchard/7045057

akorosov commented 9 years ago

@neothemachine , @mortenwh , what is the status here? I think there was a related question in #79:

Would it work to have a mapper_tests package in the root of the nansat repos and a tests module as nansat.tests which tests the core functionality of nansat (like tutorial now)? Could Travis CI then only run nansat.tests whereas we as the developers could test the mappers with the mapper_tests package?

letmaik commented 9 years ago

Sure, you can tell nosetests which directories it should include for testing. Then we would have to exclude mapper_tests from the packages attribute of setup.py which was recently replaced with the automatic find_packages().

akorosov commented 9 years ago

Continued in #79

akorosov commented 9 years ago

Nansat is installed on TravisCI using gdal and other libs from miniconda without i/o support of netCDF. Tests are run in proper time and on proper packages. Although tests fail (because miniconda version cannot read netCDF) integration can be considered to be done. Therefore I close this ticket and open another one: #102