template_tags error seen for DUNE

[dougbenjamin]

Hi,

See this error below - (template_tags function not defined in dune.py file)

10/21/2024 22:33:27.629: MoverTask[fardet-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_v09_81_00d02_merged_skip000000_lim000020_stage_makecaf_20241009T023641.root]: data file already exists at the destination and has correct size. Not overwriting 10/21/2024 22:33:27.641: Mover: Mover failed: hd-protodune_detector_run0000028005_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple_merged_skip000050_lim000014_testing_20241009T023519.root status: queued error: Traceback (most recent call last): File "/exp/dune/app/home/dunepro/venvs/declad/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run result = task.run() ^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/mover.py", line 389, in run metacat_dataset_did = custom.metacat_dataset( self.FileDesc, metadata, self.Config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 163, in metacat_dataset return rucio_dataset(desc, metadata, config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 155, in rucio_dataset meta.update(template_tags(metadata)) ^^^^^^^^^^^^^ NameError: name 'template_tags' is not defined

[marcmengel]

Closing this as fixed in #44 , which was mistagged as fixing #41...

[dougbenjamin]

After fetching latest tag 2.3.1

dunepro@dunedecladgpvm02 declad]$ git status HEAD detached at 2.3.1 Untracked files: (use "git add ..." to include in what will be committed) declad/custom/init.py

nothing added to commit but untracked files present (use "git add" to track)

10/22/2024 13:06:26.596: Mover: Mover failed: fardet-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_v09_81_00d02_merged_skip000020_lim000020_stage_makecaf_20241009T024029.root status: downloading metadata error: Traceback (most recent call last): File "/exp/dune/app/home/dunepro/venvs/declad/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run result = task.run() ^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/mover.py", line 399, in run metacat_dataset_did = custom.metacat_dataset( self.FileDesc, metadata, self.Config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 163, in metacat_dataset return rucio_dataset(desc, metadata, config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 155, in rucio_dataset meta.update(template_tags(metadata)) ^^^^^^^^^^^^^ NameError: name 'template_tags' is not defined

Should this routine be added to dune.py in custom area -

        def template_tags(metadata):
            return {}

adding the dummy routine to dune.py appears to have fixed the issue.

[marcmengel]

That should work but you're not supposed to have to add that; it should have defined it that way if the import didn't work...

[dougbenjamin]

but it did not. so ... how do we get the import to work.

here is the Run script for starting declad

!/usr/bin/env bash

Note this script is located same directory as declad.py

. $HOME/venvs/declad/bin/activate SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) cd $SCRIPT_DIR source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

setup rucio

setup metacat

setup sam_web_client export SAM_EXPERIMENT=dune export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune export PATH="/dune/app/home/dunepro/miniconda3/bin:$PATH" env | sort which python3 python3 -c "import sys; print('\n'.join(sys.path))" exec python3 -u product/declad.py -d -c config.yaml -p declad.pid $@

[dougbenjamin]

I am likely doing something wrong but can not see it. - here is PYTHONPATH - PYTHONPATH=/cvmfs/fermilab.opensciencegrid.org/products/common/db//../prd/sam_web_client/v3_6/NULL/python

[marcmengel]

So wait, you 're running 2.3.1 and still getting a "NameError: name 'template_tags' is not defined"?!?

-- Marc W. Mengel Computer Services Senior Developer

Data Management and Applications Fermi National Accelerator Laboratory 630 840 8256 office www.fnal.govhttps://email.fnal.gov/owa/redir.aspx?C=JkVsWjSyHEuWV2kFJWNPyC9OyMKf7dEIp_yxPk9yfhfadE3mdLfI6T8Ztf8BVaJjzoIe8oJ5H0M.&URL=http%3a%2f%2fwww.fnal.gov https://email.fnal.gov/owa/redir.aspx?C=JkVsWjSyHEuWV2kFJWNPyC9OyMKf7dEIp_yxPk9yfhfadE3mdLfI6T8Ztf8BVaJjzoIe8oJ5H0M.&URL=http%3a%2f%2fwww.fnal.gov Connect with us! Newsletterhttps://email.fnal.gov/owa/redir.aspx?C=-29AtpIe40SAvGF4aiNr86KvVjap7dEIuxIdg8Q5bYa9sT7QTFVhxhVnl8zwXkEuBSuAQLRkEfg.&URL=http%3a%2f%2fwww.fnal.gov%2fpub%2ftoday | Facebookhttps://email.fnal.gov/owa/UrlBlockedError.aspx | Twitterhttps://email.fnal.gov/owa/redir.aspx?C=-29AtpIe40SAvGF4aiNr86KvVjap7dEIuxIdg8Q5bYa9sT7QTFVhxhVnl8zwXkEuBSuAQLRkEfg.&URL=https%3a%2f%2ftwitter.com%2fFermilabToday

---

From: Doug Benjamin @.> Sent: Tuesday, October 22, 2024 1:22 PM To: fermitools/declad @.> Cc: Marc W Mengel @.>; State change @.> Subject: Re: [fermitools/declad] template_tags error seen for DUNE (Issue #42)

[EXTERNAL] – This message is from an external sender

I am likely doing something wrong but can not see it. - here is PYTHONPATH - PYTHONPATH=/cvmfs/fermilab.opensciencegrid.org/products/common/db//../prd/sam_web_client/v3_6/NULL/python

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fermitools_declad_issues_42-23issuecomment-2D2429955326&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=mA4qFs7Hfq-joWhWXnsXfA&m=JKSlKOSfa_B3GcLlggnLQ6W9dmxdbZljKXgcdDcRq4ph5Hwa4APpWHA1Akrk8SHH&s=jItokMxl7TJXGnbhBka_RG10iH8b9RhFMsGcU0kYSvA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAB6GV7INUEAJYSWEEENM6LZ42JURAVCNFSM6AAAAABQLRHFS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRZHE2TKMZSGY&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=mA4qFs7Hfq-joWhWXnsXfA&m=JKSlKOSfa_B3GcLlggnLQ6W9dmxdbZljKXgcdDcRq4ph5Hwa4APpWHA1Akrk8SHH&s=LLj05SwXAZFjOMYu1ZwqxQishNV8zTlMno7o99nq2sU&e=. You are receiving this because you modified the open/close state.Message ID: @.***>

[dougbenjamin]

yes hence the hack

[marcmengel]

Aaargh.. I'm not reading the stack trace right; you're getting that failure in "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py" which is the symlink to the custom/dune.py ...

[marcmengel]

Yes we need to add one there in the custom module, if we're calling it in the custom module... sigh.

[marcmengel]

It's the mover.py code that's protected against it not being in the custom module; but when I moved that rucio_dataset() code into the custom module, it then needs to be in there... Sigh.

[marcmengel]

Tagged 2.3.2

[dougbenjamin]

It works thanks