OneStop is a data discovery system being built by CIRES researchers on a grant from the NOAA National Centers for Environmental Information. We welcome contributions from the community!
[ ] Is it necessary to have try/catch around code in methods that shouldn't be waiting on an asynchronous prior run method? Ex: S3Utils.upload_s3. Then a user has to know to check the run, and what put that upload call in a while loop? What if it will never pass 'cuz of the problem that the file doesn't exist?
[ ] Should WebPublisher have onestop search methods? If we wish to have that functionality perhaps make a class more suited for it.
[ ] Verify search_onestop in WebPublishing is respecting params. Lead to removing semi-dup unused get_granules_onestop.
[ ] How to allow a user to use our application without credentials.
[ ] Hope is to have in helm values. Perhaps 2 helm values, one with credentials that will be listed in gitignore and other with unconfidential information that can be checked in.
[ ] Move any e2e type of scripts to tests/e2e directory. Perhaps pull apart into smaller chunks? Only move ...?
[ ] CsbExtractor rename to something more generic?
Random fixes:
[ ] Make the avro schema variables that are optional actually optional, currently not inside the S3MessageAdapter.transform method but are in the schemas.
[ ] Any spot upload_archive is called need to check the response is not None.
[ ] Move test_message in scripts to after where config dict is initialized.
[ ] add try/except Client error to scripts get_uuid_metadata( and read_bytes?)
[ ] Create a parent class, KafkaWrapper?, to hold register_client, connect (shared methods that are the same in both classes)
[ ] WebPublisher.publish_registry shouldn't take "method" and "metadata_type" params as a string, but instead an enum. At very least needs else statement to catch if they mistype the type or used lowercase.
[ ] Replaced prints with log statements and change any using logging to use ClientLogger.
[ ] Change logging. to self.logger
[ ] Move wait_for method in WebPublisher unit test file to tests.utils
Determine how to address these issues:
Random fixes:
[ ] Make the avro schema variables that are optional actually optional, currently not inside the S3MessageAdapter.transform method but are in the schemas.
[ ] Any spot upload_archive is called need to check the response is not None.
[ ] Move test_message in scripts to after where config dict is initialized.
[ ] add try/except Client error to scripts get_uuid_metadata( and read_bytes?)
[ ] Create a parent class, KafkaWrapper?, to hold register_client, connect (shared methods that are the same in both classes)
[ ] WebPublisher.publish_registry shouldn't take "method" and "metadata_type" params as a string, but instead an enum. At very least needs else statement to catch if they mistype the type or used lowercase.
[ ] Replaced prints with log statements and change any using logging to use ClientLogger.
[ ] Change logging. to self.logger
[ ] Move wait_for method in WebPublisher unit test file to tests.utils
[ ] Removed unused methods (collection_id_map)
[ ] Rename method def get_csv_s3(self, boto_client, bucket, key): https://github.com/cedardevs/onestop-clients/blob/master/onestop-python-client/onestop/util/S3Utils.py#L99 so directly below we have read_bytes_s3 ... so maybe we have get_s3_bytes get_s3_file read_bytes_s3 -> get_s3_bytes get_csv_s3 -> get_s3_file
[ ] In launch_e2e.py uncomment out commented out postgres chunk. requires having a container running locally that was being problematic.
[ ] Upgrade pyyaml to at least 5.4 (vulnerability https://github.com/cedardevs/onestop-clients/security/dependabot/onestop-python-client/requirements.txt/PyYAML/open)