search

sonatype-nexus-community / nexus-blobstore-google-cloud

Nexus Repository Manager Blobstore backed by Google Cloud Storage
https://help.sonatype.com/en/configuring-blob-stores.html#google-cloud-blob-store
Eclipse Public License 1.0
39 stars 16 forks source link

Is there a way to migrate from plugin version 0.4.0 to 0.10.1 #55

Closed axcosta closed 4 years ago

axcosta commented 4 years ago
nblair commented 4 years ago

Great question @axcosta. Between release 0.4.0 and 0.10, we redesigned how the "deleted blobs index" data was stored in Datastore and moved the storage of total blob count and size from a file in the bucket to Datastore. More detail on these changes can be found in https://github.com/sonatype-nexus-community/nexus-blobstore-google-cloud/blob/master/design/datastore_for_blobstore_metadata.md.

I'd still like to explore this idea with you, if you are willing.

Do you have multiple google cloud blobstores in your NXRM instance? Have you run the "compact blobstore" task successfully against the google cloud blobstore? What is the current blobstore total blob count and size? What is your tolerance for precision on the total blob count and size?

axcosta commented 4 years ago

Thanks for your prompt answer, @nblair !

Answers below:

nblair commented 4 years ago

🤔ok, this helps. I want to highlight it's fantastic to hear you're successfully proxying that many artifacts 👍 - at least I hope it's been successful :)

I'm assuming you plan to do an in-place upgrade if your existing NXRM instance? Here are some of the challenges we face:

The first fact would prevent you from being able to do an in-place upgrade with the blobstore plugin in the same GCP project. Given the size of the proxy, the time/resource to walk through all the content would be prohibitive to recalculating the sizes of the vol shards.

Is running a new NXRM instance with the latest versions in a new GCP project be an option to consider?

axcosta commented 4 years ago

@nblair , yes, it's been successful so far running NXRM in GKE at GCP. Although I think Nexus is an extremely outdated software BUT I cannot afford any paid solution at the moment.

Answering to your question, yes, I can use a new GCP project. And keep in mind that for the proxied artifacts I'm not so worried on preserving those. I'm looking really for a solution to migration of the private ones (which are very few).

Thank you again for your time on this!

nblair commented 4 years ago

@axcosta thanks for your willingness to try using a second instance, that helps us here.

Assuming that the private blobstore only has one repository, this is what I would do:

Current state

New Server

In this scenario the private repository in the new server is a proxy repository of http://oldserver/private against a new, empty bucket. As all of the artifacts are fetched through http://newserver/private, all of the content is migrated properly into the bucket. Once all content has been fetched, the oldserver can be decommissioned, as the proxy can continue to host the content. On the new server's proxy repository, you need to be aware of two things:

  1. You can stop the proxy from making additional requests to the remote once the oldserver is gone by checking "Block outbound connections on the repository".
  2. The Invalidate Cache button on http://newserver/proxy will become dangerous, as it will delete your content, and without the oldserver you would have no way to rebuild (You'd need to engage your backup strategy at that point).

In this proposed state, http://newserver/private would act just like http://oldserver/private to your clients, unless you needed to publish.

If you need to publish to this private repository on the new server, I would recommend employing a group repository type:

In this scenario, your clients consuming private artifacts from oldserver can continue to access them from the same url and path. The only difference would be for your publishers, who would need to change their publishing path to /private-publish instead of /private.

Would this work for you? By rebuilding the bucket via these combinations of repositories, they can naturally rebuild the content the way the should be through normal client activity, and we don't have to have downtime for a potentially messy migration.

nblair commented 4 years ago

Hi @axcosta! Any feedback on the above idea? I have some time blocked off tomorrow to work on the plugin, let me know if that strategy will work or if we need to explore further.

axcosta commented 4 years ago

Hi Nicholas,

Sorry not responding earlier as I have been on vacations just returning today back to work. Yes, definitely your idea worked well on our case. We have two instances running at the moment (there are artifacts which are not accessed often, so that's why we're still maintaining the old and new Nexus) being the "old" instance configured as a proxy.

Thank you very much for your help on this matter!

Best regards, Andre.

Em qua., 19 de fev. de 2020 às 11:58, Nicholas Blair < notifications@github.com> escreveu:

Hi @axcosta https://github.com/axcosta! Any feedback on the above idea? I have some time blocked off tomorrow to work on the plugin, let me know if that strategy will work or if we need to explore further.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sonatype-nexus-community/nexus-blobstore-google-cloud/issues/55?email_source=notifications&email_token=ABAPPXF5TSU5RDAD3XLVIODRDVCLFA5CNFSM4KNMIQPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMIFFLI#issuecomment-588272301, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAPPXHGFRWVV7FDWS5OM5TRDVCLFANCNFSM4KNMIQPA .

nblair commented 4 years ago

Awesome @axcosta - I'm glad to hear it worked :) Thanks for the follow up!