Error on download_log when duration too long

[dthonon]

In GitLab by @fred.lpo07 on Oct 21, 2021, 11:37

I encountered an error that make transfer crash with a too long download duration, caused by a duration too big for that Integer field. Solved by using BigInteger type.

Traceback (most recent call last):
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job                                                                        
    retval = job.func(*job.args, **job.kwargs)
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/transfer_vn.py", line 402, in full_download_1                                                                     
    short_version=(1 if cfg.json_format == "short" else 0),
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/download_vn.py", line 665, in store                                                                               
    taxo, territorial_unit_ids, short_version=short_version
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/download_vn.py", line 587, in _store_search                                                                       
    timing,
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/store_all.py", line 132, in log                                                                                   
    site, controler, error_count, http_status, comment, length, duration
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/export_vn/store_postgresql.py", line 1259, in log                                                                           
    self._conn.execute(stmt)
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1262, in execute                                                                           
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection                                                            
    self, multiparams, params, execution_options
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _execute_clauseelement                                                            
    cache_hit=cache_hit,
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context                                                                  
    e, statement, parameters, cursor, context
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1995, in _handle_dbapi_exception                                                           
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_                                                                             
    raise exception
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context                                                                  
    cursor, statement, parameters, context
  File "/data/home/geonatuser/VN_env/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute                                                                      
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range

[SQL: INSERT INTO src_vn_json.download_log (site, controler, error_count, http_status, comment, length, duration) VALUES (%(site)s, %(controler)s, %(error_count)s, %(http_status)s, %(comment)s, %(length)s, %(duration)s) RETURNING src_vn_json.download_log.id]
[parameters: {'site': 'ff', 'controler': 'observations', 'error_count': 8, 'http_status': 200, 'comment': 'ff => Iter: 20, 2139 obs, taxo_group: 1, territorial_unit: 53, date: 24/04/2021, interval: 5', 'length': 8405913, 'duration': 3649730952.082}]
(Background on this error at: http://sqlalche.me/e/14/9h9h)
2021-10-20 21:59:59,599 - ERROR - transfer_vn - The job Observations_ff crashed

[dthonon]

In GitLab by @fred.lpo07 on Oct 25, 2021, 00:11

mentioned in commit 7452f8ee59b5c17168205f8b1c5c4a2bb194d183

[dthonon]

mentioned in commit 1d3a6a49b5809c30b908c8f6468ff800ea42ab0a

[dthonon]

mentioned in commit 5a85fecf00a4bb41ff09e5c02b9db48f06c0a86c

[dthonon]

mentioned in commit b4bbffe5c31f51e374c84e9bd65a3a042b05ee26