72 user story as user i want to configure per harvest source i have want to harvest the catalog as well

[hcvdwerf]

🚀 Pull Request Checklist

• Title:

  • [ ] A brief, descriptive title for the changes.

• Description:

  • [ ] Provide a clear and concise description of your pull request, including the purpose of the changes and the approach you've taken.

• Context:

  • [ ] Why are these changes necessary? What problem do they solve? Link any related issues.

• Changes:

  • [ ] List the major changes you've made, ideally organized by commit or feature.

• Testing:

  • [ ] Describe how the changes have been tested. Include any relevant details about the testing environment and the test cases.

• Screenshots (if applicable):

  • [ ] If your changes are visual, include screenshots to help explain your changes.

• Additional Information:

  • [ ] Add any other information that might be useful for reviewers, such as considerations, discussions, or dependencies.

• Checklist:

  • [ ] I have checked that my code adheres to the project's style guidelines and that my code is well-commented.
  • [ ] I have performed self-review of my own code and corrected any misspellings.
  • [ ] I have made corresponding changes to the documentation (if applicable).
  • [ ] My changes generate no new warnings or errors.
  • [ ] I have added tests that prove my fix is effective or that my feature works.
  • [ ] New and existing unit tests pass locally with my changes.

Summary by Sourcery

Add support for configurable catalog harvesting in FairDataPointRecordProvider and update FairDataPointCivityHarvester to utilize this configuration. Enhance logging and update documentation to reflect the new configuration option. Include new unit tests to ensure the feature works as expected.

New Features:

• Introduce a configuration option to enable or disable catalog harvesting in the FairDataPointRecordProvider class.

Enhancements:

• Refactor FairDataPointCivityHarvester to support the new catalog harvesting configuration.
• Update logging format for better readability and consistency.

Documentation:

• Add documentation for the new ckanext.fairdatapoint.harvest_catalogs configuration setting in the README.md.

Tests:

• Add unit tests for the new catalog harvesting feature in FairDataPointRecordProvider.
• Add unit tests for the FairDataPointCivityHarvester to verify the new configuration handling.

[sourcery-ai[bot]]

Reviewer's Guide by Sourcery

This pull request implements a new feature to configure catalog harvesting on a per-harvest basis for the FAIR Data Point harvester. The changes include:

1. Adding a new configuration option ckanext.fairdatapoint.harvest_catalogs to enable/disable catalog harvesting globally.
2. Allowing this setting to be overridden in the harvester configuration JSON.
3. Modifying the FairDataPointRecordProvider to respect this configuration.
4. Updating the FairDataPointCivityHarvester to handle the new configuration option.
5. Adding new tests to cover the added functionality.
6. Updating the README with information about the new configuration option.

The implementation focuses on flexibility, allowing users to control catalog harvesting at both the global and per-harvest levels.

File-Level Changes

Files	Changes
ckanext/fairdatapoint/harvesters/domain/fair_data_point_record_provider.py	Added a new configuration option harvest_catalogs to FairDataPointRecordProvider
ckanext/fairdatapoint/harvesters/domain/fair_data_point_record_provider.py	Modified _process_catalog method to respect the harvest_catalogs setting
ckanext/fairdatapoint/harvesters/fair_data_point_civity_harvester.py	Updated FairDataPointCivityHarvester to handle the new harvest_catalogs configuration
ckanext/fairdatapoint/harvesters/fair_data_point_civity_harvester.py	Added a new method _get_harvest_catalog_setting to determine the harvest_catalogs value
ckanext/fairdatapoint/tests/test_record_provider.py ckanext/fairdatapoint/tests/test_fair_data_point_civity_harvester.py	Added new tests for the harvest_catalogs functionality
README.md	Updated README with information about the new ckanext.fairdatapoint.harvest_catalogs configuration option

---

Tips

- Trigger a new Sourcery review by commenting `@sourcery-ai review` on the pull request. - Continue your discussion with Sourcery by replying directly to review comments. - You can change your review settings at any time by accessing your [dashboard](https://app.sourcery.ai): - Enable or disable the Sourcery-generated pull request summary or reviewer's guide; - Change the review language; - You can always [contact us](mailto:support@sourcery.ai) if you have any questions or feedback.

[Markus92]

@sourcery-ai review