Reviewer Checklist

[ ] Required existing tests pass (ie full_test.sh, others as appropriate)
[x] NO REQUIRED unit tests (explain why not required)
[x] Required integration tests modified and pass for new functionality
[x] NO REQUIRED documentation (explain why not required)
[x] Required release notes added for new/modified functionality
[x] NO REQUIRED updates to other repos (explain why not required)

Related Issues

fixes NRLMMD-GEOIPS/geoips#624 fixes NRLMMD-GEOIPS/geoips#625

Testing Instructions

Run ./tests/integration_tests/full_install.sh to make sure your test data gets installed. Please let me know what you think about the folder renaming in check_system_requirements.sh. This was needed as some datasets installed as test_data_<ds_name>_<version_number>_github.com and we needed them to be either test_data_<ds_name> or test_data_<ds_name>_<version_number>.

Summary

Portions setup/check_system_requirements.sh test data install code was outdated due to the old instance of NextCloud being taken down recently. We've created a new instance of NextCloud which hosts a large majority of the data used for testing GeoIPS, and this required updating portions of the code which used the old links to the new link locations.

As we were making those changes, we also found that setup/download_test_data.py would not work for non .git hosted datsets. This is because a change was made to setup/check_system_requirements.sh which sent the output of the raw repsonse from requests.get to a logfile rather than piping it to tar extraction then a logfile. To fix this, we added a conditional in setup/check_system_requirements.sh which determined the source of the dataset, and either send the output of setup/download_test_data.py directly to a logfile (.git-based), or piped it to tar extraction then to a log file.

These changes address the new data locations and the bug introduced to setup/check_system_requirements.sh.

modified: geoips/commandline/ancillary_info/test_data.py
modified: geoips/commandline/geoips_config.py
modified: setup/check_system_requirements.sh
modified: tests/integration_tests/full_install.sh
modified: tests/unit_tests/commandline/cli_top_level_tester.py

Output

geoips) (base) [evan@spring integration_tests]$ ./full_install.sh 

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'git' appears to be installed successfully
    /home/evan/anaconda3/envs/geoips/bin/git

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'python' appears to be installed successfully
    /home/evan/anaconda3/envs/geoips/bin/python

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'gcc' appears to be installed successfully
    /home/evan/anaconda3/envs/geoips/bin/gcc

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'gfortran' appears to be installed successfully
    /usr/bin/gfortran

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'g++' appears to be installed successfully
    /home/evan/anaconda3/envs/geoips/bin/g++

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193607_install.log
SUCCESS: 'scipy/openblas' appear to be installed successfully

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: cartopy_shapefiles appear to be successfully installed
         if you are having comparison errors, you can try
         deleting the contents of $CARTOPY_DATA_DIR, and
         rerunning check_system_requirements.sh cartopy_shapefiles
drwxr-xr-x 2 evan evan 65536 Jan 12 16:37 /home/evan/geoips/geoips_packages/test_data/cartopy/shapefiles/natural_earth/cultural
drwxr-xr-x 2 evan evan 36864 Jan 12 16:37 /home/evan/geoips/geoips_packages/test_data/cartopy/shapefiles/natural_earth/physical

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo geoips appears to be installed successfully
    drwxr-xr-x 12 evan evan 4096 May 29 20:32 /home/evan/geoips/geoips_packages/geoips

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo data_fusion appears to be installed successfully
    drwxr-xr-x 9 evan evan 4096 May 28 15:31 /home/evan/geoips/geoips_packages/data_fusion

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo recenter_tc appears to be installed successfully
    drwxr-xr-x 9 evan evan 4096 May 15 20:18 /home/evan/geoips/geoips_packages/recenter_tc

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo geoips_clavrx appears to be installed successfully
    drwxr-xr-x 8 evan evan 4096 May 15 20:19 /home/evan/geoips/geoips_packages/geoips_clavrx

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo geoips_plugin_example appears to be installed successfully
    drwxr-xr-x 9 evan evan 4096 May 20 15:34 /home/evan/geoips/geoips_packages/geoips_plugin_example

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo template_basic_plugin appears to be installed successfully
    drwxr-xr-x 9 evan evan 4096 May 20 15:35 /home/evan/geoips/geoips_packages/template_basic_plugin

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: source repo template_fusion_plugin appears to be installed successfully
    drwxr-xr-x 9 evan evan 4096 Feb 6 19:15 /home/evan/geoips/geoips_packages/template_fusion_plugin

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: settings repo .vscode appears to be installed successfully
    drwxr-xr-x 3 evan evan 4096 Jan 12 16:55 /home/evan/geoips/geoips_packages/.vscode

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: tests data repo test_data_noaa_aws appears to be installed successfully
    drwxr-xr-x 3 evan evan 4096 Apr 4 2023 /home/evan/geoips/geoips_packages/test_data/test_data_noaa_aws

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: tests data repo test_data_amsr2 appears to be installed successfully
    drwxr-xr-x 5 evan evan 4096 Apr 4 2023 /home/evan/geoips/geoips_packages/test_data/test_data_amsr2

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: tests data repo test_data_clavrx appears to be installed successfully
    drwxr-xr-x 3 evan evan 4096 Mar 13 16:59 /home/evan/geoips/geoips_packages/test_data/test_data_clavrx

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: tests data repo test_data_gpm appears to be installed successfully
    drwxr-xr-x 5 evan evan 4096 Apr 4 2023 /home/evan/geoips/geoips_packages/test_data/test_data_gpm

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
SUCCESS: tests data repo test_data_sar appears to be installed successfully
    drwxr-xr-x 5 evan evan 4096 Apr 25 2023 /home/evan/geoips/geoips_packages/test_data/test_data_sar

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log
Installing tests data repo test_data_scat_1.11.2 .... 
  /home/evan/geoips/geoips_packages/test_data/test_data_scat_1.11.2/ from https://io2.cira.colostate.edu/s/J73tEcn22smktMi/download?path=%2F&files=test_data_scat_1.11.2.tgz via cira
DOWNLOADING: NextCloud Dataset test_data_scat_1.11.2 @ https://io2.cira.colostate.edu/s/J73tEcn22smktMi/download?path=%2F&files=test_data_scat_1.11.2.tgz
tar -xz -C /home/evan/geoips/geoips_packages/test_data >> /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193608_install.log 2>&1
SUCCESS: Decompressed test_data_scat_1.11.2

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193632_install.log
Installing tests data repo test_data_scat_1.11.3 .... 
  /home/evan/geoips/geoips_packages/test_data/test_data_scat_1.11.3/ from https://io2.cira.colostate.edu/s/J73tEcn22smktMi/download?path=%2F&files=test_data_scat_1.11.3.tgz via cira
DOWNLOADING: NextCloud Dataset test_data_scat_1.11.3 @ https://io2.cira.colostate.edu/s/J73tEcn22smktMi/download?path=%2F&files=test_data_scat_1.11.3.tgz
tar -xz -C /home/evan/geoips/geoips_packages/test_data >> /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193632_install.log 2>&1
SUCCESS: Decompressed test_data_scat_1.11.3

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193650_install.log
SUCCESS: tests data repo test_data_smap appears to be installed successfully
    drwxr-xr-x 5 evan evan 4096 Apr 4 2023 /home/evan/geoips/geoips_packages/test_data/test_data_smap

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193650_install.log
SUCCESS: tests data repo test_data_viirs appears to be installed successfully
    drwxr-xr-x 4 evan evan 4096 Aug 14 2023 /home/evan/geoips/geoips_packages/test_data/test_data_viirs

Install log: /home/evan/geoips/geoips_packages/outdirs/logs/20240530.193650_install.log
SUCCESS: tests data repo test_data_fusion appears to be installed successfully
    drwxr-xr-x 3 evan evan 4096 May 8 2023 /home/evan/geoips/geoips_packages/test_data/test_data_fusion

NRLMMD-GEOIPS / geoips

624 update nextcloud links #626

Reviewer Checklist

Related Issues

Testing Instructions

Summary

Output