Reprocess entire mission for commissioning

[mgalloy]

Reprocess the data and publish the data on the web.

Need to start this by November 22, 2023 to give enough to time, with a buffer to complete by AGU.

Reprocess 20210715 until the cameras were changed in November 2022.

• 20210715-20221109 (including end date)

Tasks

• [x] delete current installation
• [x] update production ucomp-config
• [x] release version 1.0
• [x] free space on dawn
• [x] move aside old engineering directory
• [x] remove alternate personal ucomp-pipeline installations
• [x] start reprocessing 2022 first
• [x] release version 1.0.1
• [x] start reprocessing
• [x] fix up database plots with #164

[mgalloy]

We can free up space in the various process.* directories:

- [ ] 35G   process.bilinear
- [ ] 1.3T  process.centering-comparison
- [ ] 1001G process.commissioning-test
- [x] 139G  process.commissioning-test-2
- [ ] 188G  process.commissioning-test-3
- [ ] 176G  process.commissioning-test-4
- [ ] 1.3G  process.commissioning-test-5
- [ ] 44G   process.diff-centering
- [ ] 391M  process.fail-to-process
- [ ] 1.4G  process.gbu-test
- [ ] 34G   process.he-contsub
- [ ] 1.8T  process.intermediate
- [ ] 78G   process.l1-verification
- [x] 813G  process.latest
- [x] 185G  process.linewidth
- [ ] 37G   process.no-badframes
- [ ] 59G   process.nomask
- [ ] 55G   process.offsets
- [ ] 84G   process.publish
- [ ] 33G   process.raw-chi2
- [ ] 6.6G  process.rcam
- [ ] 29G   process.regression
- [ ] 6.6G  process.tcam
- [ ] 0         process.unit
- [ ] 264G  process.wilson
- [x] 338G  process.workshop
- [x] 122G  process.workshop-test
6.7T    total

[mgalloy]

We have 392 days with data that we are reprocessing. Currently it is all at version 0.5.0 (from the Workshop reprocessing) except for one day that I did as a test day for Don.

0.5.0: 391 days
0.5.1-dev: 1 day

I will report progress on the reprocessing in this issue.

Two servers were knocked out of the pool of processors for this reprocessing because they couldn't see the production database, but Tom S. was able to get them seeing it again. So we should have 8 servers with 2 processes (16 total processes) each running most of the time.

[mgalloy]

Started reprocess at approximately 3:50 pm on Wed Nov 22, 2023.

[jburkepile]

YEAH MIKE!

[mgalloy]

42 days finished after about 5 hours:

logs$ progress
Wed Nov 22 20:56:28 MST 2023
--L--: 12 days
0.5.0: 337 days
0.5.1-dev: 1 day
1.0.0: 42 days

[jburkepile]

That is fantastic!

[mgalloy]

Almost 17 hours in, and about 25% done:

$ progress
Thu Nov 23 08:39:15 MST 2023
--L--: 12 days
-----: 2 days
0.5.0: 280 days
1.0.0: 98 days

[mgalloy]

We are 30 hours in and 42% complete (predicting a finish on Saturday evening, but there are always a few stragglers at the end to clean up).

$ progress
Thu Nov 23 21:55:47 MST 2023
--L--: 15 days
-----: 2 days
0.5.0: 211 days
1.0.0: 164 days

[mgalloy]

We are about 40:40 into the reprocessing and 54% complete (still predicting a Saturday evening finish).

$ progress
Fri Nov 24 08:31:21 MST 2023
--L--: 13 days
-----: 3 days
0.5.0: 166 days
1.0.0: 210 days

[mgalloy]

We are 48 hours into the reprocessing and 60% complete (probably an early Sunday morning finish now).

$ progress
Fri Nov 24 15:51:05 MST 2023
--L--: 14 days
-----: 2 days
0.5.0: 142 days
1.0.0: 234 days

[mgalloy]

We are 53.5 hours into the reprocessing and 67% complete (probably an early Sunday morning finish).

$ progress
Fri Nov 24 21:14:03 MST 2023
--L--: 14 days
-----: 2 days
0.5.0: 115 days
1.0.0: 261 days

[mgalloy]

We are under 65 hours into the reprocessing and 80% complete (still projecting an early Sunday morning finish).

$ progress
Sat Nov 25 08:36:54 MST 2023
--L--: 11 days
-----: 2 days
0.5.0: 67 days
1.0.0: 312 days

[mgalloy]

We are just under 78.5 hours into the reprocessing and over 96% complete — we should finish just after midnight.

$ progress
Sat Nov 25 22:15:20 MST 2023
--L--: 6 days
-----: 2 days
0.5.0: 6 days
1.0.0: 378 days

[mgalloy]

We are basically done, just picking up a couple of stragglers:

$ progress
Sun Nov 26 08:35:12 MST 2023
--L--: 1 day
-----: 2 days
0.5.0: 1 day
1.0.0: 388 days

For some reason, 20220524 and 20220525 are not marked done in the database, but logs show they processed without error. They don't have much data, but there are some science files processed. 20210821 is almost done and then 20210822 will be processed when that is done.

[mgalloy]

The missing versions on 20220524 and 20220525 are from this:

# From issue #159: "the Cropico battery died, and all temps were misreported as
# being ~4C too low. This looks to get bad 20220523.212656.81.ucomp.789.l0.fts
# and remains bad for the rest of the day. We didn't collect UCoMP data again
# until the 30th, and the problem was resolved on the 25th.
[20220523.212656]
process                      : NO

[20220530]
process                      : YES

[mgalloy]

The last day finished around 10:50 am this morning (91 hours total, but almost all days were done last night after 78 hours or so).

[mgalloy]

Started fixing up the database plots at 11:24 am on Sunday 26 November 2023.

[mgalloy]

Started reprocessing with version 1.0.01 at 4:05 pm on Wednesday 29 November 2023.

[mgalloy]

We are nearly 5 hours into the reprocessing and just under 7% complete.

$ progress
Wed Nov 29 21:57:47 MST 2023
--L--: 13 days
-----: 2 days
1.0.0: 350 days
1.0.1: 27 days

[mgalloy]

We are just over 16 hours into the 1.0.1 reprocessing and about 18.5% complete (projected Sunday morning finish).

$ progress
Thu Nov 30 08:17:08 MST 2023
--L--: 15 days
-----: 2 days
1.0.0: 302 days
1.0.1: 73 days

[mgalloy]

I've updated my progress script:

$ progress
Thu Nov 30 21:49:08 MST 2023 (running for 29 hours)
--L--: 11 days
-----: 2 days
1.0.0: 233 days
1.0.1: 146 days
estimated completion time: Sat Dec  2 23:54:00 MST 2023

[mgalloy]

$ progress
Fri Dec  1 07:52:26 MST 2023 (running for 39 hours)
--L--: 8 days
-----: 2 days
1.0.0: 201 days
1.0.1: 181 days
estimated completion time: Sun Dec  3 06:14:00 MST 2023

[mgalloy]

$ progress
Fri Dec  1 21:49:02 MST 2023 (64% complete, running for 53 hours)
estimated completion time: Sun Dec  3 03:40:00 MST 2023

--L--: 16 days
-----: 2 days
1.0.0: 124 days
1.0.1: 250 days

[mgalloy]

$ progress
Sat Dec  2 07:20:21 MST 2023 (75% complete, running for 63 hours)
estimated completion time: Sun Dec  3 03:50:00 MST 2023

--L--: 12 days
-----: 2 days
1.0.0: 84 days
1.0.1: 294 day

[mgalloy]

$ progress
Sat Dec  2 22:19:32 MST 2023 (93% complete, running for 78 hours)
estimated completion time: Sun Dec  3 04:06:00 MST 2023

--L--: 9 days
-----: 2 days
1.0.0: 18 days
1.0.1: 363 days

[mgalloy]

We are done with the 1.0.1 reprocessing. Now, I will start the jobs for fixing up the plots from the database for #164.