owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
78 stars 21 forks source link

✨ cli: etl pr #3177

Closed lucasrodes closed 1 month ago

lucasrodes commented 1 month ago

This amends https://github.com/owid/etl/pull/3152 to replace etl d pr with etl pr. Also, it updates the documentation accordingly.

etl d draft-pr is left for backwards compatibility. Might remove it once the team adopts etl pr.

owidbot commented 1 month ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-cli-pr

chart-diff: ✅ No charts for review.
data-diff: ```diff 2024-08-22 10:04:58 [error ] Traceback (most recent call last): File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json return complexjson.loads(self.text, **kwargs) File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads return _default_decoder.decode(s) File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode obj, end = self.raw_decode(s) File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/owid/etl/etl/datadiff.py", line 423, in cli lines = future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/owid/etl/etl/datadiff.py", line 416, in func differ.summary() File "/home/owid/etl/etl/datadiff.py", line 254, in summary self._diff_tables(self.ds_a, self.ds_b, table_name) File "/home/owid/etl/etl/datadiff.py", line 122, in _diff_tables table_a = future_a.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f return copy(f, *args, **kw) File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__ do = self.iter(retry_state=retry_state) File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter result = action(retry_state) File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in self._add_action_func(lambda rs: rs.outcome.result()) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__ result = fn(*args, **kwargs) File "/home/owid/etl/etl/datadiff.py", line 837, in get_table_with_retry return ds[table_name] File "/home/owid/etl/etl/datadiff.py", line 278, in __getitem__ return tables.load() File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 312, in load return self.iloc[0].load() # type: ignore File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 363, in load return Table.read(uri) File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 178, in read table = cls.read_feather(path, **kwargs) File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 350, in read_feather cls._add_metadata(df, path, **kwargs) File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 322, in _add_metadata metadata = cls._read_metadata(path) File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 384, in _read_metadata return cast(Dict[str, Any], requests.get(metadata_path).json()) File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 978, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) = Dataset garden/who/2024-05-20/vehicles = Table vehicles ⚠ Found errors, create an issue please Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-08-22 10:05:01 UTC Execution time: 281.66 seconds