search

sot / kadi

Chandra commands and events
https://sot.github.io/kadi
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

Stop pushing to ftp in regular processing #134

Closed taldcroft closed 5 years ago

taldcroft commented 5 years ago

Using lucky as a gateway to push commands and events to the GRETA network has been overtaken by pulling from GRETA via rsync. This is more reliable and simpler.

Note that task_schedule_occ_{cmds,events}.cfg are no longer running on GRETA since moving to Wayside.

taldcroft commented 5 years ago

I just checked the rsync log to explicitly verify that files are getting transferred as expected.

2019:214:06:00:03.log:data/cmd_states/cmd_states.db3
2019:214:06:00:03.log:data/cmd_states/cmd_states.h5
2019:214:06:00:03.log:data/kadi/cmds.h5
2019:214:06:00:03.log:data/kadi/cmds.pkl
2019:216:06:00:02.log:data/kadi/events.db3
jeanconn commented 5 years ago

What are your thoughts about how this would be installed? Just rsync as aca user to update the ska2 task schedules?

taldcroft commented 5 years ago

Installed. For some reason the change in #121 was never installed, but now it is.

kadi$ diff task_schedule_cmds.cfg /proj/sot/ska/data/kadi/task_schedule_cmds.cfg 
43c43,52
<       exec update_cmds --data-root=$ENV{SKA_DATA}/kadi
---
>       exec update_cmds --ftp --data-root=$ENV{SKA_DATA}/kadi
>       <check>
>         <error>
>           #    File           Expression
>           #  ----------      ---------------------------
>              kadi.log     error
>              kadi.log     warning
>              kadi.log     fatal
>         </error>
>       </check>
kadi$ diff task_schedule_events.cfg /proj/sot/ska/data/kadi/task_schedule_events.cfg 
46c46
<       exec update_events --data-root=$ENV{SKA_DATA}/kadi/update_events
---
>       exec update_events --ftp --data-root=$ENV{SKA_DATA}/kadi/update_events

aca-kadi% rsync -v task_schedule_events.cfg /proj/sot/ska/data/kadi/task_schedule_events.cfg
task_schedule_events.cfg

sent 2934 bytes  received 31 bytes  5930.00 bytes/sec
total size is 2846  speedup is 0.96

aca-kadi% rsync -v task_schedule_cmds.cfg /proj/sot/ska/data/kadi/task_schedule_cmds.cfg

sent 2353 bytes  received 31 bytes  4768.00 bytes/sec
total size is 2267  speedup is 0.95
taldcroft commented 5 years ago

And of course the issue with #121 just highlights the unreliable nature of the Ska2 architecture for task_schedule and out-of-environment files.

jeanconn commented 5 years ago

I think it just means that if there's still a Makefile, we should probably use it or remove it. That's still an issue in Ska3 like with this package. We can also use conda to install non-Python files with the Makefile, but I get the impression you like setup.py better for that in general.

taldcroft commented 5 years ago

Sorry, I meant "out-of-environment" files (e.g. /proj/sot/ska/data/kadi/task_schedule_cmds.cfg). About conda and Makefile, are you talking about a post-install hook to run a Makefile? Anyway, any mechanism that installs out-of-environment files in conda is the wrong direction and unreliable.

I realize that Ska3 does not strictly enforce the package-based architecture just yet, but that is the plan with time and effort. One package at a time, e.g. cheta will be next.

jeanconn commented 5 years ago

I thought the issue here was that "make install" was not run in ska2 for kadi 3.15.4. I thought that was in-environment.

Regarding "About conda and Makefile, are you talking about a post-install hook to run a Makefile?" I don't know what you mean. For packages with a makefile, having the build.sh just run "make install" seems fine, so that's not a post-install step, that's just a package build step.

taldcroft commented 5 years ago

I thought the issue here was that "make install" was not run in ska2 for kadi 3.15.4.

I don't have a clue, so the plan is to fix process so memory is not required.

About conda and Makefile, I guess that works in the sheltered world of a production environment (assuming you set SKA to the install root before running make in the build), but fails for a standalone where SKA != SKA_ARCH_OS. If you think about how it was (is) in Ska2, $SKA/share is outside the installed environment which lives in $SKA_ARCH_OS. In Ska3, share is defined to live in $SKA_ARCH_OS, but legacy task schedule or other legacy scripts assume $SKA/share, which is (in general) outside the environment.

Takeaway: for Ska3 we should not use our custom Makefile that installs files based on $SKA. As packages are migrated to Ska3, the Makefile should be removed.

jeanconn commented 5 years ago

I think our old include logic wrt to SKA / SKA_ARCH_OS doesn't make sense for Makefile installs, but PREFIX should be reliable on the conda side.

But I'm not arguing to keep Makefiles, I'm just saying that if we have to jump through any other hoops to get setup.py to do something reasonable, we can examine this as needed.