Dynamic locking Merritt

[dloy]

Current

Currently at least two subsystem locks use the presents of a file to determine if the lock is in place:

• Ingest pause lock
• storage large queue lock

The advantage of this feature is the simplicity for adding and deleting a file for setting the lock. And, the presence or non-presence of a file is a straight forward test.

The difficulty of continuing to use this method is that:

• it requires a pre-existing directory system
• micro-services are moving to environments where a directory system may not exist or if it does exist it is not commonly available to other micro-services for access (i.e. Docker)

Requirements

• must be relatively easy to set from a distributed program environment
• must be easy to test from a distributed program environment
• must be cheap

Discussion

Two broad categories similar to this issue are

• feature flag - dynamically determining if a feature in a service is available or not (or partial ...)
• dynamic configuration - being able to change the functioning on a process based on configurations that can be remotely set

Several tools fall into these categories

• commercial includes LaunchDarkly (uses commercial subscription and complex)
• AWS AppConfig - has both request and response costs that become excessive based on frequency of tests, also complex.

Many tools provide a variety of complex features that are dynamically available to programs. Some of these features use webpages for updating. The commercial tools are expensive at either a subscription basis or on a quantity of access basis. They tend to be far more complex than what we require.

For our current environment this handling is possible through different services:

• SSM
• Zookeeper
• S3

SSM

Advantages:

• Accessible in our current programming environment
• Able to be set manually

Disadvantages:

• Setting requires a special authorized environment

Zookeeper

Advantages:

• Currently used
• Can be set programmatically
• Has been used specifically for this purpose by Netflix
• Accessible from flag programs requiring this flag

Disadvantages: - needs more investigation

• Appears to require a continual refresh to keep flag in queue - this should be investigated
• Requires new program work

S3

Advantages:

• Easily updateable and accessible from existing code
• globally accessible

Disadvantages:

• Requires new program work
• May require new Java SDK jars in programs using
• May require specialized node/bucket for handling

Proof of concept

I am currently working on a simplified library for proof of concept using S3. Using a special jar for this handling will bring in the AWS jars needed for S3 handling.

[mreyescdl]

Design details if implemented as a ZKlock

Access method Consumer/onHold() is where the check for a HOLD file exists. This is the top level of where the switch to ZK will be.

Changes in ZK library

• Need the support for a peek() method in DistributedLock. Currently, we only have create and destroy.
• The peek method will check for the existence of a lock
• Example new lock and node will be: "merrittServiceLocks/accessLarge.lock"

Changes in Access

• The Consumer daemon already accesses ZK queue to grab content for processing. We'll need to add support for a ZK lock.
• Each Access host will know which lock node to check though SSM var
• Within the onHold() method, check will call the peek() method for ZK to see if it exists. If so, then go into loop until lock is cleared. Eventually, all threads will be put into a loop state until lock is cleared.

Changes in Ingest

• Support for creation and destruction of Service locks in Admin Manager

[dloy]

I tried using a slightly different approach using Mark's locking mechanism that currently exists in the inventory system. This approach uses the presence of a specific znode as the flag. The znode is similar to a file/directory that always contains content.

With this approach the apache zookeeper methods can then be used for what would be many file operation.

• zookeeper.create - create a znode
• zookeeper.exists - does znode already exist
• zookeeper.delete - delete this znode
• zookeeper.getData - get content of znode

By using CreateMode.PERSISTENT in the create operation, the znode remains after the zookeeper session, or service has terminated. The current inventory locks for arks uses CreateMode.EPHEMERAL that terminates the znodes with the zookeeper session. It appears that only the zookeeper.delete will remove the znode - not affected by zookeeper.exists.

I have run a variety of tests:

• test on/off same server - local zookeeper
• test on/off different client/target programs same server - local zookeeper
• test on/off different client/target programs same server - run zookeeper - kill zookeeper - run zookeeper
• test on/off different client/target programs - different hosts - stage zookeeper

All tests indicate that servers on the same zookeeper can detect the flag and changes to the flag - even if zookeeper has been shutdown and restarted and the detection server has been restarted.

I must admit my knowledge of the details on using zookeeper as a property service is sketchy! So far these tests have indicated that zookeeper can provide a properties file system that can persist over shutdowns of either the client or zookeeper service.

[dloy]

I put together a prototype/proof of concept version that runs from a zootoken branch for storage.

[dloy]

flag

Prototype operations:

http://<store server>/flag/[status|state|set|clear]/<service>/<flag name>?t=[xml | json | anvl]

flag state:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/state/access/LargeAccessHold?t=xml'

flag status:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/status/access/LargeAccessHold?t=xml'

flag set large:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/set/access/LargeAccessHold?t=xml'

flag clear large:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/clear/access/LargeAccessHold?t=xml'

flag set small:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/set/access/SmallAccessHold?t=xml'

flag clear small:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/clear/access/SmallAccessHold?t=xml'

flag set 1 level:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/set/access?t=xml'

flag clear 1 level:
curl -X POST 'http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/flag/clear/access?t=xml'

zoodata

To set znode content example:

op=$1
curl -L -X POST \
-F "content=this is my data" \
-F "responseForm=xml" \
 "http://uc3-mrtsandbox2-stg.cdlib.org:35121/storage/zoodata/${op}/access/LargeAccessHold

Jersey:

   @POST
    @Path("zoodata/{operation}/{service}/{flagName}")
    @Consumes(MediaType.MULTIPART_FORM_DATA)
    public Response callSetZooData(
            @PathParam("operation") String operation,
            @PathParam("service") String service,
            @PathParam("flagName") String flagName,
            @DefaultValue("") @FormDataParam("content") String payload,
            @DefaultValue("xhtml") @FormDataParam("responseForm") String formatType,
operation: [status|state|set|clear]
service: here - access
flagName - znode path
payload: data saved in znode 
formatType: xml, json, anvl

tok:zooTokenStat response

<?xml version="1.0" encoding="UTF-8"?>
<tok:zooTokenState xmlns:tok='http://uc3.cdlib.org/ontology/mrt/store/tok'>
    <tok:data>this is my data</tok:data>
    <tok:tokenStatus>true</tok:tokenStatus>
    <tok:runDate>2022-10-24T12:35:36-07:00</tok:runDate>
    <tok:zooProcess>state</tok:zooProcess>
    <tok:zooConnectionString>uc3-mrtzk01x2-stg.cdlib.org:2181,uc3-mrtzk02x2-stg.                        cdlib.org:2181,uc3-mrtzk03x2-stg.cdlib.org:2181</tok:zooConnectionString>
    <tok:zooFlagPath>access/LargeAccessHold</tok:zooFlagPath>
    <tok:zooNodeBase>/mrt.lock</tok:zooNodeBase>
    <tok:lockCreated>2022-10-24T12:35:21-07:00</tok:lockCreated>
</tok:zooTokenState>

data: content of znode
tokenStatus: true=flag exists; false=no flag
runDate: current run date
zooProcess: state | status | set | clear
zooConnectionString: zookeeper connection string
zooFlagPath: znode key
zooNodeBase: base for all znode properties and flags
lockCreated: lock creation data if present