YangCatalog / module-compilation

Tools to collect, extract, and validate YANG modules
https://yangcatalog.org/private-page
Apache License 2.0
0 stars 5 forks source link

Override compilation results in redis on populate #264

Closed bskqd closed 1 year ago

bskqd commented 1 year ago

Running the compile_modules.py script again after a run of the populate.py script will not help because statistics and other generated files won't be overwritten because of lack of information for them (or managing their upload will not be straightforward at all as we have to manage different ietf arguments and they all have specific logic), so I decided to check the /var/yang/all_modules directory inside the compile_modules.py script, so if the file is already in the /var/yang/all_modules and the hash of the file that is being processed at the moment is different from the hash of the file inside the /var/yang/all_modules, we will compile the file from the /var/yang/all_modules dir, as it's always the actual file. I didn't remove updating of Redis from the compile_modules.py script because if this case happens, then the actual compilation data will be set to Redis after the next run of compile_modules.py

resolves #248

github-actions[bot] commented 1 year ago

Coverage report

Full coverage report summary (click to unfold) check_archived_drafts.py: 91%
create_config.py: 100%
extractors/__init__.py: 100%
extractors/draft_extractor.py: 67%
extractors/extract_elem.py: 77%
extractors/helper.py: 44%
figures_and_stats/__init__.py: 100%
figures_and_stats/yang_get_stats.py: 90%
ietf_modules_extraction/__init__.py: 100%
ietf_modules_extraction/gather_ietf_dependent_modules.py: 81%
ietf_modules_extraction/yang_version_1_1.py: 75%
job_log.py: 77%
main_page_generation/__init__.py: 100%
main_page_generation/private_page.py: 52%
message_factory/__init__.py: 100%
message_factory/message_factory.py: 26%
metadata_generators/__init__.py: 100%
metadata_generators/extract_emails.py: 67%
modules_compilation/__init__.py: 100%
modules_compilation/file_hasher.py: 90%
parsers/__init__.py: 100%
parsers/yang_parser.py: 65%
redis_connections/__init__.py: 100%
redis_connections/constants.py: 100%
redis_connections/redis_connection.py: 96%
redis_connections/redis_user_notifications_connection.py: 61%
rename_file_backup.py: 78%
tests/test_check_archived_drafts.py: 96%
tests/test_extract_elem.py: 96%
tests/test_extract_emails.py: 94%
tests/test_file_hasher.py: 98%
tests/test_gather_ietf_dependent_modules.py: 98%
tests/test_get_stats.py: 97%
tests/test_private_page.py: 100%
tests/test_remove_directory_content.py: 98%
tests/test_rename_file_backup.py: 97%
tests/test_utility.py: 100%
tests/test_yang_version_1_1.py: 97%
utility/__init__.py: 100%
utility/static_variables.py: 100%
utility/utility.py: 85%
versions.py: 100%

The coverage rate went from 83.46% to 83.52% :arrow_up:

100% of new lines are covered.

Diff Coverage details (click to unfold) ### utility/utility.py `100%` of new lines are covered (`85.6%` of the complete file).
richardzilincikPantheon commented 1 year ago

I also thought of doing it this way at first, but it occurred to me that maybe we want to have the data for the different versions of the file in the html/json reports. For example here https://yangcatalog.org/private/CiscoXR672YANGPageCompilation.html at the top of the page we claim that these are results for modules in https://github.com/YangModels/yang/tree/main/vendor/cisco/xr/672/, which might not be the case anymore. I don't really know which is better though, this also makes sense in a different way and is probably fine too.

bskqd commented 1 year ago

I also thought of doing it this way at first, but it occurred to me that maybe we want to have the data for the different versions of the file in the html/json reports. For example here https://yangcatalog.org/private/CiscoXR672YANGPageCompilation.html at the top of the page we claim that these are results for modules in https://github.com/YangModels/yang/tree/main/vendor/cisco/xr/672/, which might not be the case anymore. I don't really know which is better though, this also makes sense in a different way and is probably fine too.

Oh, I see, not sure what will be the right choice in this situation, @SlavomirMazurPantheon what do you think about it?

P.S. I believe it's better to keep it as it is implemented now, so we have the same data in Redis and compilation files