Feature: ability to pull remote configuration files over http

[simojo]

closes https://github.com/AstuteSource/chasten/issues/17

This PR allows for users to specify a Url directive to a text file for the --config argument. Additionally, it tests all of the newly added features. It also moves around some functions to reduce ambiguity of the names of methods (e.g. moving things related to configuration to configuration.py)

Example usage:

chasten analyze test --config https://raw.githubusercontent.com/AstuteSource/chasten-configuration/master/config.yml --search-path .

Summary of all changes:

• configuration.py
  • configuration.display_configuration_directory() (moved)
  • Moved from main.py to configuration.py, because this is specific to configuration.
  • configuration.validate_checks_file() (new)
  • New method that extracts logic specific to checks file into a function
  • configuration.validate_configuration_files() (moved, modified)
  • Moved from main.py to configuration.py, because this is specific to configuration.
  • Modified to account for configuration files being pathlib.Path and urllib3.Url
  • configuration.convert_configuration_text_to_yaml() (new)
  • New method that breaks out functionality of larger method into a function.
  • configuration.extract_configuration_details_from_config_url() (new)
  • New method to distinguish extracting configuration details from URL versus Path
  • configuration.extract_configuration_details_from_config_dir() (new)
  • New method to distinguish extracting configuration details from URL versus Path
• validate.py (moved, modified)
  • validate.validate_file()
  • Moved from main.py to validate.py, because this is specific to validation
• util.py
  • util.is_url() (new)
  • New method to determine if string is valid URL
• tests/
  • test_checks.py (modified)
  • JSON_SCHEMA_COUNT modified to fill in rest of schema from canonical schema found in validate.py
  • test_main.test_cli_analyze_url_config() (new)
  • Makes sure CLI correctly handles simple URL config.
  • test_main.test_cli_analyze_url_config_with_local_checks_file() (new)
  • Confirm that using the command-line interface aborts execution when given a URL config that uses a local file path to specify checks files.
  • test_main.test_cli_analyze_url_config_with_url_checks_file() (new)
  • Confirm that using the command-line interface correctly handles a local config that references URL endpoints for each checks file.
  • test_main.test_cli_analyze_local_config_with_url_and_local_checks_files() (new)
  • Confirm that using the command-line interface correctly handles a local config that references a combination of URL endpoints and local files for each checks file.
  • test_main.test_cli_analyze_url_config_with_url_and_local_checks_files() (new)
  • Confirm that using the command-line interface aborts execution when given a URL config that references a combination of URL endpoints and local files for each checks file.
  • test_util.test_is_url_correct() (new)
  • Use Hypothesis to confirm that URLs are correctly recognized/unrecognized.
• .chasten
  • .chasten/config_url_and_local_checks_files.yml (new)
  • Local config used in testing.
  • .chasten/config_url_and_checks_file.yml (new)
  • Local config used in testing.
• README.md (edited)
  • Modified to indicate that URLs are allowed as valid configurations.

[simojo]

This is working to its fullest extent. The only remaining task is to create a test that does a sample run of chasten using a raw.githubusercontent file from the chasten-configuration directory.

[simojo]

I have finished the test cases that verify this PRs functionality. The only remaining step is to merge https://github.com/AstuteSource/chasten-configuration/pull/1, as it is integral to testing the pulling of remote configuration files.

[simojo]

Coverage appears increased (73% -> 74%)

---------- coverage: platform linux, python 3.11.4-final-0 -----------
Name                       Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------
chasten/__init__.py            0      0      0      0   100%
chasten/__main__.py            2      2      0      0     0%   3-5
chasten/checks.py             51      6     28      5    84%   24, 86->89, 94, 99-101, 117
chasten/configuration.py     140     17     38      8    86%   157-162, 167, 203, 246-249, 346, 386-389, 407-410, 418-421, 431-433
chasten/constants.py         121      0     20      0   100%
chasten/database.py           75     62     24      0    13%   42-49, 56-76, 92-101, 115-128, 144-267, 272-289
chasten/debug.py              10      0      0      0   100%
chasten/enumerations.py       13      0      0      0   100%
chasten/filesystem.py         99     38     26      4    63%   138->145, 151->156, 162->167, 186-206, 220-236, 247-288, 294-302, 308-315
chasten/main.py              196     54     74     13    71%   76-87, 145-152, 154->exit, 163, 184->191, 347-350, 359-362, 407, 464->exit, 505->509, 510->509, 562, 570-577, 587, 628-669, 717-733, 787-804, 816-825, 832-834
chasten/output.py             93     24     30      2    71%   80-81, 87-88, 94-96, 102, 108-126, 132-134, 143-150, 204->168, 213->212
chasten/process.py            38      9     16      1    70%   27-43, 109
chasten/results.py            46      0      0      0   100%
chasten/server.py             29     18      4      0    33%   23-36, 43-67
chasten/util.py               54      7     14      2    87%   36-37, 46, 63-64, 116-117
chasten/validate.py           31      3      6      3    84%   118, 144, 162
----------------------------------------------------------------------
TOTAL                        998    240    280     38    74%
Coverage JSON written to file coverage.json

[boulais01]

@simojo I went ahead and resolved merge conflicts - I tried to ensure all functionality was preserved, but it might be worth taking a quick look to make sure I didn't accidentally delete something you needed. The conflicts were in util.py and test_util.py

[simojo]

Thanks @boulais01. I've fixed the linting errors associated with that merge. Everything should be good now.

[simojo]

Hi @simojo, please consider using the PR template next time. That would be really helpful. Thanks.