Re-queuing and access

[dloy]

Zookeeper was introduced to the Access processing to allow a server separation between the creation of large and small content. By analyzing the size of content to be processed the access consumer will direct the processing to a large or small queue for handling.

Startup process:

• get API request by access to build a container file from an object
• build an S3 token containing the content used for the archive file creation
• call to consumer thread to pass request to appropriate server for processing

A separate thread retrieves the queued request and begins the process:

• retrieve zookeeper request
• determine if the process has already begun using token
• start creation of container file
• move created container file to S3
• reset the S3 token to indicated final processing status
• reset zookeeper request status as success or fail

The zookeeper queue in this process is ONLY used for directing content to the proper server (small or large). Locking of the queue is available for preventing content failures during server down time (e.g. deploy).

"re-queuing" in Dryad was available but zookeeper was not involved - the request was resubmitted from the beginning.

The only way re-queuing on zookeeper will work is if no processing of the archive file has started or if the container processing has successfully completed. All intermediate states are not restartable.

For the new zookeeper handling all failures on a zookeeper entry should be treated as a delete .failure. Alternatively, mark all processing of the entry as .success. No effort should be used for restarting using zookeeper. The Dryad approach works well - allow the resubmission of the initial request. This avoids any possible state problems for collision on data content which is a serious issue for S3 content.

[dloy]

Forcing unlock after both success() and fail() conditions using the latest mrt-zk allowed the admin tool to requeue the zookeeper entry.

For this to be successful:

• original access run fails while building container file (zip or tar)
• failure condition no longer occurs (e.g. S3 unable in first run to find content for zip is successful in subsequent retry)

This process does not include:

• server dies while processing leaving content locked
• failure before queued by zookeeper (404 on object)