`tagObject` produces an orphaned data object

[doulikecookiedough]

In the below proposed scenario raised by @artntek , tagObject could possibly produce an orphaned data object.

Thread A                    Thread B
========                    ========
   :                            :
uploads object X          uploads object X
 with pidX                   with pidX
   :                            :
(gets lock)                 (awaiting lock)
   :                            :
tagging obj X,                  :
FAILS ❌                        :
(returns cid lock)              :
   :                        (gets cid lock)
(returns pid lock)               :
   :                        (gets pid lock)
(awaiting lock)                 :
   :                        TAGS obj X ✅
   :                        (returns lock)
(gets lock)                     :
UNTAGS obj X ✅                 :
(returns lock)                  :
   :                            :
   :                            :
               DONE

In the example process shown, to add context, there are two paths in which the client could take. 1) storeObject(InputStream, pid, ...) where the pid is locked and tagObject is automatically called after 2) storeObject(InputStream), with no lock, then tagObject is called verifying the data object is valid

In Scenario 1)

• The client would call storeObject with a given pid, meaning that it would hit the first synchronization lock on the pid
• If Thread A and Thread B are both executed at the same time with this method signature, one request will be rejected as the synchronization in this flow will throw an exception if the pid is already being used
  • One pid can only ever reference one data object
  • The synchronization guarantees that every request has to acquire the lock, check the collection/array, then proceed onwards
• In this process, there is only one thread we are concerned about, and if an unexpected exception occurs, the tagging process is rolled back

In Scenario 2) (your described example)

• The client would store the data object first, then validate the data object (-> deleteIfInvalidObject before calling `tagObject)
• If the tagging process fails in Thread A, since Thread B is in a waiting state and is notified immediately once a lock is released, it would proceed to lock the cid, and then pid.
• Thread B would then attempt to tag the pid-cid combo as expected. Depending on where the failure happened, Thread B will finish the tagging process based on the status of the reference files, and release the lock
• Thread A will then untag the pid-cid combo based on the status of the reference files, and release the lock
• An orphaned data object is produced (Obj X)

Possible Solution:

Similar to how storeObject rejects duplicate store requests for a given pid, perhaps tagObject should also do the same. If a pid is in the process of being tagged (it's locked), any subsequent requests to tag a pid should be rejected.

There can only ever be 1 pid reference file, meaning the pid should only ever appear once in all the cid reference files as well.

Since we first lock the cid, then proceed to lock the pid, it sounds reasonable to reject a tagObject request if the pid is already in the process of being tagged.

We may have to re-organize the order, where we attempt to lock the pid first.

[doulikecookiedough]

To resolve the process, unTagObject will be extracted and moved into storeHashStoreRefsFiles. Expected exceptions will be re-thrown, and unexpected exceptions will trigger unTagObject before releasing the lock and bubbling up.

Thread A                    Thread B
========                    ========
   :                            :
uploads object X          uploads object X
 with pidX                   with pidX
   :                            :
(gets cid lock)         (awaiting lock)
(gets pid lock)                 :
   :                            :
tagging obj X,                  :
FAILS ❌                        :
catches exception               :
UNTAGS obj X ✅                 :
   :                            :
(returns cid lock)              :
   :                        (gets cid lock)
(returns pid lock)              :
   :                        (gets pid lock)
   :                            :
   :                        TAGS obj X ✅
   :                        (returns lock)
               DONE

[doulikecookiedough]

This has been completed via Bug-97: tagObject Produces Orphaned Object