Getting this PR up because it has ind/build changes. ~Will add more context later this am.~ But is tied to #231 and things @padilla410 discovered in #248
update
The focus of this PR is to make working on new temperature data sources more developer-friendly, by clearing out several of the gotchas in target connections and triggers that in hindsight, explain a number of mystery builds from the past.
I've pulled in changes from Julie's open PR so that we're not colliding on indicator file changes. I will keep up with updates or merge of that PR if things change and get them into this PR too. Pulling in changes from an open PR is generally not a practice we'd do, but in scipiper/shared cache repos, it is sometimes necessary.
I moved the "forever stale" target from #242 to 1_fetch so it could be used by other steps easily
I added the stale trigger file to two targets in 6_coop, which include indexing the files on google (was triggered by a dummy string in the past) and the coop_wants target that relies on the index ind. It is necessary to pair targets to use the trigger file, otherwise it doesn't work correctly (this explains why coop_wants uses it too)
since the google drive index will happen with every build that depends on it now and that call to google is expensive and we could get API-limited if accessing too often(?), I put in a time check on the indicator for the file index (see below for freshened: 2021-12-10 05:44:22 -0600) and it won't re-index google drive unless the last build was less than trigger_wait minutes ago (I set this to 30). This seems a little questionable, but I feel confident it is a major improvement over what was happening in the past
the NTL targets were removed, which was to address #248. Those are in the in folder anyhow and we don't need to re-download them. I did go ahead and update one of the files since a newer version was available, so that explains the file name and parser change in here
I linked the downloaded file index to the 7a_ munge task table, since those weren't connected before. Likely another source of confusing double builds that happened in the past.
Added NULL as default on the trigger file targets, so leaving out that arg will skip the trigger stale step in case you want to avoid this when developing.
Nice fixes to some questionable choices I made 🙈! Just a few comments -- I still don't understand why you need the always stale trigger in both targets? (but I see your note, so trust you've thought it through)
Getting this PR up because it has ind/build changes. ~Will add more context later this am.~ But is tied to #231 and things @padilla410 discovered in #248
update
The focus of this PR is to make working on new temperature data sources more developer-friendly, by clearing out several of the gotchas in target connections and triggers that in hindsight, explain a number of mystery builds from the past.
coop_wants
target that relies on the index ind. It is necessary to pair targets to use the trigger file, otherwise it doesn't work correctly (this explains whycoop_wants
uses it too)freshened: 2021-12-10 05:44:22 -0600
) and it won't re-index google drive unless the last build was less thantrigger_wait
minutes ago (I set this to 30). This seems a little questionable, but I feel confident it is a major improvement over what was happening in the pastin
folder anyhow and we don't need to re-download them. I did go ahead and update one of the files since a newer version was available, so that explains the file name and parser change in hereNULL
as default on the trigger file targets, so leaving out that arg will skip the trigger stale step in case you want to avoid this when developing.