Update to Python 3.12

[wmwv]

Google Cloud Functions will deprecate Python 3.7 in 2024 January. Use this motivation to update to Python 3.12.

Steps

1. [x] Update deploy.sh scripts to specify --runtime python312 instead of --runtime python37.
2. [x] Test cloud functions locally to check requirements.
3. [x] Check requirements.txt files to see if there are extra pins.
4. [x] Test deployment in test framework on GCP.

[wmwv]

Pins:

• requirements.txt

  [...]
  astropy==3.2.1
  [...]

• ./broker/cloud_functions/classify_snn/requirements.txt

  [...]
  supernnova==1.3
  [...]
  # pip install torch doesn't work because cloud fncs don't support GPUs.
  # instead, install cpu version from wheel
  # https://stackoverflow.com/questions/55449313/google-cloud-function-python-3-7-requirements-txt-makes-deploy-fail
  https://download.pytorch.org/whl/cpu/torch-1.9.0%2Bcpu-cp37-cp37m-linux_x86_64.whl

  This one for supernnova makes sense. We do want to specify the version of the classifier explicitly. But we should check in on pytorch.

• broker/night_conductor/requirements.txt

[...]
apache-beam[gcp]==2.25.0
astropy==3.2.1
fastavro==0.22.3
# apache-beam 2.25.0 requires pyarrow<0.18.0,>=0.15.1
# but google-cloud-bigquery>=2.10.0 requires pyarrow>=3
google-cloud-bigquery<2.10
google-cloud-core>==1.4.1
google-cloud-dataflow
google-cloud-datastore>==1.15
google-cloud-storage==1.38.0
iminuit==1.4.9
matplotlib==3.1.1
# older pyarrow versions require numpy<1.20 to successfully convert datatypes
# gcp_utils needs this to load a pandas dataframe to bigquery
# this is a numpy bug: https://github.com/numpy/numpy/issues/17913
numpy==1.19.5
pandas==0.24.2
[...]
sncosmo==2.2.0
[...]

Why does the night_conductor need sncosmo?

• broker/cloud_functions/ps_to_gcs/requirements.txt

  fastavro==1.4.4  # fastavro==1.4.4 is known to work for ztf (see below)
  google-cloud-logging
  google-cloud-storage==1.38.0
  pgb-broker-utils>=0.2.46

Plus two in docs/source/working-notes/troyraen which we can ignore.

[troyraen]

Thanks for posting these. broker/night_conductor/requirements.txt definitely looks bloated. Looks more like past-me just copy/pasted some other requirements.txt file I was using and never cleaned it up. I can't think of any reason it should need sncosmo, and definitely not apache-beam[gcp].

[troyraen]

But, I recommend not spending any time on night_conductor w.r.t. this issue. It is troubled (it runs the metadata collector), but it works right now so I try not to touch it. If we are going to do any work on it, we should replace it rather than fix it. See #171.

[wmwv]

Ah, okay. Restrict this ticket to only things that need Cloud Functions (Cloud Run).

[troyraen]

Which is:

• broker/broker_utils/
• broker/cloud_functions/

[troyraen]

FWIW, from my notes about the astropy pin, night_conductor requirements, etc.

pip3 install pgb-broker-utils==0.2.28 &> install_broker_utils_0.2.28.out
# this does not succeed
# astropy requires jinja2 which requires MarkupSafe.soft_unicode
# but this was removed in v2.1.0
# https://markupsafe.palletsprojects.com/en/2.1.x/changes/#version-2-1-0

[troyraen]

@hernandezc1 Is this complete? I think it is, just checking.