search

bcgov / entity

ServiceBC Registry Team working on Legal Entities
Apache License 2.0
23 stars 59 forks source link

Migrate Notebook Report to GCP #24360

Open bolyachevets opened 1 week ago

bolyachevets commented 1 week ago

https://github.com/bcgov/sbc-auth/tree/main/jobs/notebook-report

the job seems to generate/email manager reports from sbc-auth database. will need to migrate to poetry and set up CI/CD github actions for it.

stevenc987 commented 1 week ago

The following error was from notebook job on sbc-auth. The errors was caused when it was trying to connect to auth database on openshift. We have migrated auth database from openshift to gcp now so this job wouldn't work on openshift anymore. Please note that this job only run on prod only

ERROR!!!

Traceback (most recent call last):

  File "notebookreport.py", line 114, in processnotebooks

    pm.execute_notebook(os.path.join(notebookdirectory, file), data_dir+'temp.ipynb', parameters=None)

  File "/usr/local/lib/python3.8/site-packages/papermill/execute.py", line 134, in execute_notebook

    raise_for_execution_errors(nb, output_path)

  File "/usr/local/lib/python3.8/site-packages/papermill/execute.py", line 241, in raise_for_execution_errors

    raise error

papermill.exceptions.PapermillExecutionError:

Exception encountered at "In [4]":

[0;31m---------------------------------------------------------------------------[0m

[0;31mNameError[0m                                 Traceback (most recent call last)

Cell [0;32mIn[4], line 3[0m

[1;32m      1[0m filename [38;5;241m=[39m os[38;5;241m.[39mpath[38;5;241m.[39mjoin(os[38;5;241m.[39mgetcwd(), [38;5;124mr[39m[38;5;124m'[39m[38;5;124mdata/[39m[38;5;124m'[39m)[38;5;241m+[39m[38;5;124m'[39m[38;5;124mauth_weekly_statstill[39m[38;5;124m'[39m [38;5;241m+[39m datetime[38;5;241m.[39mstrftime(datetime[38;5;241m.[39mnow()[38;5;241m-[39mtimedelta([38;5;241m1[39m), [38;5;124m'[39m[38;5;124m%[39m[38;5;124mY-[39m[38;5;124m%[39m[38;5;124mm-[39m[38;5;132;01m%d[39;00m[38;5;124m'[39m) [38;5;241m+[39m[38;5;124m'[39m[38;5;124m.csv[39m[38;5;124m'[39m

[0;32m----> 3[0m df [38;5;241m=[39m [43mstat_accounts_weekly_completed[49m[38;5;241m.[39mDataFrame()

[1;32m      4[0m [38;5;28;01mwith[39;00m [38;5;28mopen[39m(filename, [38;5;124m'[39m[38;5;124mw[39m[38;5;124m'[39m) [38;5;28;01mas[39;00m f:

[1;32m      5[0m     df[38;5;241m.[39minsert([38;5;241m0[39m, [38;5;124m"[39m[38;5;124mWeekly Number of New Accounts:[39m[38;5;124m"[39m, [[38;5;124m'[39m[38;5;124m'[39m], [38;5;28;01mTrue[39;00m)

 

[0;31mNameError[0m: name 'stat_accounts_weekly_completed' is not defined

Please see details at: https://sbc-registries.atlassian.net/browse/ITOPS-147751

stevenc987 commented 2 days ago

@bolyachevets Is someone working on this ticket?

bolyachevets commented 2 days ago

@bolyachevets Is someone working on this ticket?

No. It was not prioritized