search

njcuk9999 / apero-drs

A PipelinE to Reduce Observations - The DRS for SPIRou (CFHT)
MIT License
12 stars 1 forks source link

0.7.290 issue with apero_reject.py #789

Open larnoldgithub opened 2 days ago

larnoldgithub commented 2 days ago

A test with apero_reject.py failed.

apero_reject.py --identifier=3036202o,3036203o,3036204o,3036235o --test=True --autofill="1,1,1,OBJECT_grade5_24BQ03-Aug16"

20:55:27.763- |REJECT| 20:55:27.774- |REJECT| 20:55:27.774- |REJECT| SPIROU @PID-00017291985269974710-MS7R (V0.7.290) 20:55:27.775- |REJECT| py3.9.16 git:v0.7.290-stable-test 20:55:27.775- |REJECT| 20:55:27.785- |REJECT| 20:55:27.794- |REJECT| █████╗ ██████╗ ███████╗██████╗ ██████╗
20:55:27.800- |REJECT| ██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔═══██╗ 20:55:27.806- |REJECT| ███████║██████╔╝█████╗ ██████╔╝██║ ██║ 20:55:27.811- |REJECT| ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║ 20:55:27.817- |REJECT| ██║ ██║██║ ███████╗██║ ██║╚██████╔╝ 20:55:27.823- |REJECT| ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
20:55:27.828- |REJECT| 20:55:27.837-|REJECT| DRS Setup: 20:55:27.843-|REJECT| 20:55:27.844-|REJECT| DRS_ROOT: /apero/apero-drs-v0.7.290/apero 20:55:27.844-|REJECT| DRS_DATA_RAW: /data/spirou4/arlinks 20:55:27.844-|REJECT| DRS_DATA_REDUC: /data/spirou4/apero-data/offline290/red 20:55:27.844-|REJECT| DRS_DATA_WORKING: /data/spirou4/apero-data/offline290/tmp 20:55:27.844-|REJECT| DRS_CALIB_DB: /data/spirou4/apero-data/offline290/calib 20:55:27.845-|REJECT| DRS_TELLU_DB: /data/spirou4/apero-data/offline290/tellu 20:55:27.845-|REJECT| DRS_DATA_ASSETS: /data/spirou4/apero-data/offline290/asset 20:55:27.845-|REJECT| DRS_DATA_OUT: /data/spirou4/apero-data/offline290/out 20:55:27.845-|REJECT| DRS_DATA_MSG: /data/spirou4/apero-data/offline290/msg 20:55:27.845-|REJECT| DRS_DATA_RUN: /data/spirou4/apero-data/offline290/run 20:55:27.845-|REJECT| DRS_DATA_PLOT: /data/spirou4/apero-data/offline290/plot 20:55:27.846-|REJECT| DRS_DATA_OTHER: /data/spirou4/apero-data/offline290/other 20:55:27.846-|REJECT| LBL_PATH: /data/spirou4/apero-data/offline290/lbl 20:55:27.846-|REJECT| DRS_CONFIG: /apero/config/offline290/user_config.ini 20:55:27.846-|REJECT| DRS_CONFIG: /apero/config/offline290/user_constants.ini 20:55:27.846-|REJECT| DRS_CONFIG: apero.constants.default.default_config 20:55:27.846-|REJECT| DRS_CONFIG: apero.constants.default.default_keywords 20:55:27.847-|REJECT| DRS_CONFIG: config.instruments.spirou.default_config.py 20:55:27.847-|REJECT| DRS_CONFIG: config.instruments.spirou.default_keywords.py 20:55:27.847-|REJECT| DRS_CONFIG: core.instruments.default.default_constants.py 20:55:27.847-|REJECT| DRS_CONFIG: core.instruments.spirou.default_constants.py 20:55:27.847-|REJECT| DATABASE: MYSQL 20:55:27.847-|REJECT| DATABASE-calib: aperooffline290@localhost:calib_offline290_db 20:55:27.847-|REJECT| DATABASE-tellu: aperooffline290@localhost:tellu_offline290_db 20:55:27.848-|REJECT| DATABASE-findex: aperooffline290@localhost:findex_offline290_db 20:55:27.848-|REJECT| DATABASE-log: aperooffline290@localhost:log_offline290_db 20:55:27.848-|REJECT| DATABASE-astrom: aperooffline290@localhost:astrom_offline290_db 20:55:27.848-|REJECT| DATABASE-lang: aperooffline290@localhost:lang_offline290_db 20:55:27.848-|REJECT| DATABASE-reject: aperooffline290@localhost:reject_offline290_db 20:55:27.848-|REJECT| PRINT_LEVEL: all 20:55:27.849-|REJECT| LOG_LEVEL: all 20:55:27.849-|REJECT| DRS_PLOT: 0 20:55:27.854- |REJECT| 20:55:27.860-|REJECT| Arguments used: 20:55:27.866-|REJECT| 20:55:27.866-|REJECT| --IDENTIFIER: 3036202o,3036203o,3036204o,3036235o 20:55:27.866-|REJECT| --OBJNAME: None 20:55:27.866-|REJECT| --AUTOFILL: 1,1,1,OBJECT_grade5_24BQ03-Aug16 20:55:27.866-|REJECT| --TEST: True 20:55:27.872- |REJECT| ***** 20:55:29.359-!!|REJECT| E[01-010-00001]: Unhandled error has occurred: Error <class 'AttributeError'> 20:55:29.360-!!|REJECT| 20:55:29.361-!!|REJECT| Traceback (most recent call last): 20:55:29.361-!!|REJECT| File "/apero/apero-drs-v0.7.290/apero/core/utils/drs_startup.py", line 433, in run 20:55:29.361-!!|REJECT| llmain = func(recipe, params) 20:55:29.361-!!|REJECT| File "/apero/apero-drs-v0.7.290/tools/bin/apero_reject.py", line 114, in main 20:55:29.361-!!|REJECT| drs_reject.add_file_reject(params, recipe, identifier) 20:55:29.362-!!|REJECT| File "/apero/apero-drs-v0.7.290/apero/tools/module/listing/drs_reject.py", line 79, in add_file_reject 20:55:29.362-!!|REJECT| google_sheet = gspd.spread.Spread(sheet_id) 20:55:29.362-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/spread.py", line 126, in init 20:55:29.362-!!|REJECT| self.client = Client(user, config, scope, creds) 20:55:29.363-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/client.py", line 96, in init 20:55:29.363-!!|REJECT| self._root = self._drive_request(file_id="root", params={"fields": "name,id"}) 20:55:29.363-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/client.py", line 197, in _drive_request 20:55:29.363-!!|REJECT| res = self.request(method, url, params=params, json=data) 20:55:29.364-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/util.py", line 292, in request 20:55:29.364-!!|REJECT| return ClientV4.request(client, *args, kwargs) 20:55:29.364-!!|REJECT| AttributeError: type object 'Client' has no attribute 'request' 20:55:29.364-!!|REJECT| 20:55:29.474-|REJECT| 20:55:29.487-@!|REJECT| W[40-003-00005]: Recipe apero_reject has NOT been successfully completed 20:55:29.500-|REJECT|

njcuk9999 commented 2 days ago

This is the same error with the gspread function

20:55:29.364-!!|REJECT| return ClientV4.request(client, *args, kwargs)
20:55:29.364-!!|REJECT| AttributeError: type object 'Client' has no attribute 'request'

Can you check which conda environment you were in and your gspread-pandas and gspread functions?

larnoldgithub commented 1 day ago 
 offline290 Fri Oct 18 15:31:03 spdrs@apikipiha: ~
>>   pip show gspread
Name: gspread
Version: 6.1.3
Summary: Google Spreadsheets Python API
Home-page: 
Author: 
Author-email: Anton Burnashev <fuss.here@gmail.com>
License: 
Location: /conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages
Requires: google-auth, google-auth-oauthlib
Required-by: gspread-pandas
 offline290 Fri Oct 18 15:31:47 spdrs@apikipiha: ~
>>   pip show gspread-pandas
Name: gspread-pandas
Version: 3.2.2
Summary: A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.
Home-page: https://github.com/aiguofer/gspread-pandas
Author: Diego Fernandez
Author-email: aiguo.fernandez@gmail.com
License: BSD
Location: /conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages
Requires: decorator, google-auth, google-auth-oauthlib, gspread, pandas
Required-by:

Another version 290 installed in a container and succefully used to add object to the obj_db

spdrs@32dcd4761891:/opt/apero-drs$ pip show gspread
Name: gspread
Version: 5.12.4
Summary: Google Spreadsheets Python API
Home-page: 
Author: 
Author-email: Anton Burnashev <fuss.here@gmail.com>
License: 
Location: /usr/local/lib/python3.9/dist-packages
Requires: google-auth, google-auth-oauthlib
Required-by: gspread-pandas
spdrs@32dcd4761891:/opt/apero-drs$ pip show gspread-pandas
Name: gspread-pandas
Version: 3.2.2
Summary: A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.
Home-page: https://github.com/aiguofer/gspread-pandas
Author: Diego Fernandez
Author-email: aiguo.fernandez@gmail.com
License: BSD
Location: /usr/local/lib/python3.9/dist-packages
Requires: decorator, google-auth, google-auth-oauthlib, gspread, pandas
Required-by:

Ok so it seems the Offline 290 is using the 6.3.1. gspread and apero_reject.py fails. I used apero_reject.py with the version using gspread 5.1.4 and it worked, I could upload odometers to the db.

Is it possible to downgrade the gspread package from 6.3.1 to 5.1.4 in the offline environnement without corrupting my offline290 data and setup? @njcuk9999 @cusher I'm currently running the offline290, preprocessing of all SPI files that will be completed by tuesday or wednesday next week, so it's lilely better to wait before touching anything of the apero-env-v0.7.290 on apikipiha. :)