search

CDLUC3 / mrt-doc

Documentation and Information regarding the Merritt repository
8 stars 4 forks source link

[INGEST] Queue redesign (Part 3) - Support additional zookeeper states #1441

Open mreyescdl opened 1 year ago

mreyescdl commented 1 year ago

We need additional states for Ingest entries e.g. recording, processing. This is for Job queue only.

elopatin-uc3 commented 7 months ago

Latest notes: https://github.com/CDLUC3/mrt-doc/blob/main/design/queue-2023/states.md

mreyescdl commented 2 months ago

Initial work for supporting multiple states is working on Ingest Stage under the "queue_LIB" branch. Scale testing is complete. Edge case testing is not.

I'll start work on supporting HOLD and LOCK which is under ticket https://github.com/CDLUC3/mrt-doc/issues/1883 This will further test the ZK library. Once this work is complete I'll circle back to this ticket for more testing

mreyescdl commented 2 months ago

There was an issue with Ingest Unit test involving the method ingestState.setServiceName(serviceNameS); This method is instantiated when creating a ProcessManager from the Unit Test. The error involves Zookeeper Session expiration when querying if the Ingest Queue is frozen. Now that the thaw/frozen state is stored in ZK and not the file system, this is a ZK call. The Unit test tries to make this call, but gets the expiration exception.

For now, Ingest is fault tolerant and assumes this exception is from the Unit Test. In ProcessManager.java...

                try {
                   zooKeeper = new ZooKeeper(queueConnectionString, sessionTimeout, new Ignorer());
                } catch (Exception e) {
                   // Unit test catch
                   unitTest = true;
                }

@terrywbrady Can we discuss a more permanent fix?