Add Google Cloud Storage (GCS) Accessor

[mbazos]

Would like to add a Google Cloud Storage Accessor for Shedlock. This should be possible because of the guarantees here https://cloud.google.com/storage/docs/consistency also there have been other libraries that have popped up but are specific to a GCS implementation. Shedlock has some nice features and would be nice if we could add this as a provider.

I actually went ahead and did some of the coding on this but really got stuck on the testing part. Unfortunately google doesn't offer a GCS emulator that you could use in conjunction with testcontainers.org and I tried https://github.com/fsouza/fake-gcs-server and while it did work it couldn't delete files from a bucket which is needed to test the lock/unlock functionality.

Before sending in a PR I wanted just some general guidance on this and if there are any suggestions on how to go about testing this properly. I could mock the Storage class but that might not really provide the level of testing needed on a library like this. Any suggestions are welcome.

package net.javacrumbs.shedlock.provider.googlecloudstorage;

import net.javacrumbs.shedlock.core.ClockProvider;
import net.javacrumbs.shedlock.core.LockConfiguration;
import net.javacrumbs.shedlock.support.AbstractStorageAccessor;
import net.javacrumbs.shedlock.support.annotation.NonNull;

import java.time.Instant;
import java.util.HashMap;
import java.util.Map;

import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;

class GoogleCloudStorageAccessor extends AbstractStorageAccessor {
    private static final String LOCK_FILE_CONTENT = "_lock";
    private static final String MIME_TYPE_TEXT_PLAIN = "text/plain";

    private final Storage storage;
    private final String bucketName;
    private final String lockFilename;

    GoogleCloudStorageAccessor(Storage storage, String bucketName, String lockFilename) {
        this.storage = storage;
        this.bucketName = bucketName;
        this.lockFilename = lockFilename;
    }

    @Override
    public boolean insertRecord(@NonNull LockConfiguration lockConfiguration) {

        //Getting the object using `doesNotExist` will ensure a global lock
        Storage.BlobTargetOption blobOption = Storage.BlobTargetOption.doesNotExist();
        Blob blob = storage.create(blobInfo(lockConfiguration), LOCK_FILE_CONTENT.getBytes(), blobOption);

        return blob != null;
    }

    private BlobInfo blobInfo(LockConfiguration lockConfiguration) {
        Map<String, String> metadata = new HashMap<>();
        metadata.put("lockedBy", getHostname());
        metadata.put("lockName", lockConfiguration.getName());
        metadata.put("now", ClockProvider.now().toString());
        metadata.put("lockUntil", lockConfiguration.getLockAtMostUntil().toString());

        BlobId blobId = BlobId.of(bucketName, lockFilename);

        return BlobInfo.newBuilder(blobId)
            .setMetadata(metadata)
            .setContentType(MIME_TYPE_TEXT_PLAIN)
            .build();
    }

    @Override
    public boolean updateRecord(@NonNull LockConfiguration lockConfiguration) {
        Blob blob = storage.get(blobInfo(lockConfiguration).getBlobId());

        if (isLockedBy(blob)) {
            storage.create(blobInfo(lockConfiguration));
            return true;
        }

        return false;
    }

    @Override
    public boolean extend(LockConfiguration lockConfiguration) {
        Blob blob = storage.get(blobInfo(lockConfiguration).getBlobId());

        if (isLockedBy(blob) && lockConfiguration.getLockAtMostUntil().isAfter(Instant.now())) {
            storage.create(blobInfo(lockConfiguration));
            return true;
        }
        return false;
    }

    private boolean isLockedBy(Blob blob) {
        return getHostname().equalsIgnoreCase(blob.getMetadata().get("lockedBy"));
    }

    @Override
    public void unlock(@NonNull LockConfiguration lockConfiguration) {
        storage.delete(blobInfo(lockConfiguration).getBlobId());
    }
}

[lukas-krecan]

Hi, thanks a lot for contributing. I generally do not like to accept Lock Providers without automated tests as it makes it impossible to maintain the package. I do not have enough time to do the manual testing.

We can do the same as with CosmosDB integration. If you are willing to, just create a GitHub project and release the package. I will be happy to link the provider from ShedLock documentation. Is it OK for you?

[mbazos]

Thanks @lukas-krecan for the reply. I will talk to my team and see what they want to do I totally agree something like this cannot really be properly tested without automated tests and I really like the integration tests you have setup in ShedLock.

Once I talk to my team I will let you know which way we decide to go.

[lukas-krecan]

Hi, I have checked the pricing and it should be possible to test it against the real GCS. So if you will make the test work, I can provide an account for CI pieline.

[mbazos]

@lukas-krecan that would be great, yeah I think if you stay within the "free tier" we should be good. Let me work on the integration test using my own GCS bucket and once I have all the tests working I can send in a PR which then you could hook up your GCS account and your CI pipeline to validate everything is good.

It's not urgent for me to finish this up, but will probably send you a PR over the next couple weeks when I have some free time.

[lukas-krecan]

Ok, thanks

[kagkarlsson]

Did you ever publish a GCS-provider @mbazos ?

[mbazos]

I am sorry I didn't push it yet, I wanted to do this but we ended up going a different route for this specific work use-case I have.

[mbazos]

@kagkarlsson all that needs to be done is testing the above code with an actual GCP project, I submitted this back in 2021 I am not sure if there is a viable emulator which can be run in a container for GCS bucket which would be ideal. If not @lukas-krecan mentioned he could setup a GCS bucket for tests.

If you wanted to get going with this take the code above and use your own private GCS bucket for building the tests and then maybe submit the PR to @lukas-krecan and he could setup a GCS bucket for running the tests for ShedLock

[kagkarlsson]

Ok, thanks for the update. If we end up going this route I will 👍