Task `osmtgmod.import-osm-data` sometimes fails

[nesnoj]

Apparently cloning in a subprocess (osmTGmod in this case) isn't very stable. In my case it worked after the 3rd attempt. A quick research revealed that it could be related to HTTP/2 .

Note: Using the native subprocess (not the custom one from egon-data) worked out of the box, but that could also be a coincidence and it's all related to the machine or connection...

Full log:

[2024-01-10 11:24:41,555] {taskinstance.py:1150} ERROR - 
| Command '['git', 'clone', '--single-branch', '--branch',
| 'features/egon', 'https://github.com/openego/osmTGmod.git']' returned
| non-zero exit status 128.
| Output:
| Errors:
|   Klone nach 'osmTGmod' …
|   error: RPC fehlgeschlagen; curl 92 HTTP/2 stream 0 was not closed
|   cleanly: CANCEL (err 8)
|   fatal: Die Gegenseite hat unerwartet abgebrochen.
|   fatal: zu frühes Dateiende
|   fatal: index-pack fehlgeschlagen
Traceback (most recent call last):
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/__init__.py", line 202, in skip_task
    result = super(type(task), task).execute(*xs, **ks)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/venv/lib/python3.8/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/datasets/osmtgmod/__init__.py", line 56, in import_osm_data
    subproc.run(
  File "/storage/eGon-data-sh-mastr-zenodo-fix-ja/code/src/egon/data/subprocess.py", line 60, in run
    raise CalledProcessError(
egon.data.subprocess.CalledProcessError: 
| Command '['git', 'clone', '--single-branch', '--branch',
| 'features/egon', 'https://github.com/openego/osmTGmod.git']' returned
| non-zero exit status 128.
| Output:
| Errors:
|   Klone nach 'osmTGmod' …
|   error: RPC fehlgeschlagen; curl 92 HTTP/2 stream 0 was not closed
|   cleanly: CANCEL (err 8)
|   fatal: Die Gegenseite hat unerwartet abgebrochen.
|   fatal: zu frühes Dateiende
|   fatal: index-pack fehlgeschlagen
[2024-01-10 11:24:41,558] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=egon-data-processing-pipeline, task_id=osmtgmod.import-osm-data, execution_date=20240110T101336, start_date=20240110T102415, end_date=20240110T102441
[2024-01-10 11:24:45,202] {local_task_job.py:102} INFO - Task exited with return code 1

[nesnoj]

Workaround: Clone manually