search

cedardevs / onestop

OneStop is a data discovery system being built by CIRES researchers on a grant from the NOAA National Centers for Environmental Information. We welcome contributions from the community!
GNU General Public License v2.0
42 stars 20 forks source link

Onestop-clients update scripts #1507

Open erinreeves opened 3 years ago

erinreeves commented 3 years ago

Summary

- [x] Update script constructors where used to take config location and now takes a dict - [x] Changed constructor parameters of metadata_type to classname_metadata_type so could have two metadata_types in same config if needed by user.
As a user
I want to be able to execute the scripts, in the scripts directory
So that I can perform the different supported actions onestop-clients has to offer.

Additional Tasks: Due to being unable to run the full OneStop stack locally I had to execute the scripts on bastion(cedar-devs). This required building an image of onestop-python-client, onestop-s3-handler, and onestop-sme to hub.docker so I could install that on bastion and have it pull the image down and deploy it to an EKS node (which is not able to see local files on bastion).

erinreeves commented 3 years ago

Need to revisit image names

erinreeves commented 3 years ago

2021-06-05 03:30:20.364 INFO 1 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams : stream-client [psi-registry-ad13de8e-a5f2-4c4a-95e4-daa0a3746902] State transition from RUNNING to ERROR 2021-06-05 03:30:20.366 ERROR 1 --- [-StreamThread-1] o.c.o.kafka.common.util.KafkaHelpers : Streams app entered a bad state: ERROR 2021-06-05 03:30:20.367 ERROR 1 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams : stream-client [psi-registry-ad13de8e-a5f2-4c4a-95e4-daa0a3746902] All stream threads have died. The instance will be in error state and should be closed. 2021-06-05 03:30:20.367 INFO 1 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread : stream-thread [psi-registry-ad13de8e-a5f2-4c4a-95e4-daa0a3746902-StreamThread-1] Shutdown complete

erinreeves commented 3 years ago

Pull request https://github.com/cedardevs/onestop-clients/pull/58