search

scientist-softserv / adventist_knapsack

Apache License 2.0
2 stars 0 forks source link

šŸ› BUG: New Tenant #646

Open DiemBTran opened 2 years ago

DiemBTran commented 2 years ago

As of 2022-11-28, there is no wildcard subdomain and this needs to be setup in DNS. This was communicated to the client in Slack.

Ever since this commit, a user is unable to create a new tenant without causing a 500 error, locally and in staging.

ref:

https://notch8.slack.com/archives/CQK3DBHAR/p1643668927495469

Acceptance Criteria

documentation for cert: Adventist notes:

DiemBTran commented 2 years ago

Related MRs:

fix new account creation with sidekiq queue splitting !164 Site specs !165 udpate readme with new tenant create ingress requirements !181 add-test-tenant-url-to-ingress-and-tls-for-client-verification !190

DiemBTran commented 2 years ago

Rob Feb 2022:

should be resolved by both !165 (merged) and !164 (merged)

DiemBTran commented 2 years ago

Diem March 2022

Needs further review:

Summary: Was able to create a tenant (https://118-test.s2.adventistdigitallibrary.org/), but cannot access it (This site canā€™t be reached 118-test.s2.adventistdigitallibrary.orgā€™s server IP address could not be found.). There is no Sentry error. Is that an issue related to the ticket?

Details:

Was able to create tenant: https://118-test.s2.adventistdigitallibrary.org/

Image

Cannot access it-- This site canā€™t be reached 118-test.s2.adventistdigitallibrary.orgā€™s server IP address could not be found

Image

no Sentry error

Image

DiemBTran commented 2 years ago

Lea Ann May 2022

The tenants are created but an SSL cert needs to be generated for the domain. Software Services does not have access to dns server so any ssl certs need to be per specific domain. If a new tenant needs to be created, it will require a developer to update the ingress settings for that specific domain before the tenant is created.

April Rieger May 2022

SSL cert && nginx ingress

Lea Ann June 2022

To be clear, new tenants can be created successfully. They cannot be accessed until and SSL cert is generated. If a new tenant is to be created, a ticket to add SSL cert should be created for this work

DiemBTran commented 2 years ago

Katharine Van Arsdale July 2022

I'd like to test and verify this ticket, but I need to confirm the steps to create a new tenant. Can someone give me those steps? I need a reminder, since this process hasn't worked so I haven't tried it in a long time. Once I have some instructions, I understand that I can do most of the process within our instance of Hyku, with the exception of the final step, which requires me to create an SSL certification ticket every time I try to generate a new tenant.

DiemBTran commented 2 years ago

April Rieger - July 2022

Hi @KatharineV ! I went ahead and made a pdf with screenshots of how to test tenant creation here for you attached below. adventist-create-new-tenant-instructions.pdf

I went through the process of adding the new tenant to the values file in order to generate an ssl cert and ingress.

There is one more additional step, we either need to add a wildcard cert for https://s2.adventistdigitallibrary.org & https://b2.adventistdigitallibrary.org and the process will stay the same where anytime a tenant is created we would need to intervene on our end and add any subsequent tenants to the values file.

There is an alternate route that we can pursue but we would need cert-manager access to your DNS(AWS Route 53), cert-manager needs to be able to add records to Route53 in order to solve the DNS01 challenge in order to create an SSL cert dynamically. Here are a set of instructions to follow in doing so: https://cert-manager.io/docs/configuration/acme/dns01/route53/

Please let me know if you have any additional questions or need any additional information.

Thank you

DiemBTran commented 2 years ago

Katharine Van Arsdale - August 2022

Just chiming in to add that the ADL team discussed automatic SSL certifications, and this is the route we want to go if possible. Thanks, Eric, for collaborating with Software Services to set up our system to generate new tenants without human intervention! Software Services folks, if we run into any snags pursuing this route the way we have things set up, let us know.

DiemBTran commented 2 years ago

April R - Sept 2022

Thank you! Just getting back to this now.

DiemBTran commented 2 years ago

Katharine Van Arsdale - Oct 2022

Hey folks, we'd like to create a new tenant on staging and development. Is this ticket situation resolved to the point where we can do that and create new tenants in both locations? Thanks!!

Image

Update to say that I was able to create a tenant in S2, but it isn't accessible, as you can see in the screenshot error message. I can't test creating a tenant in B2 because my user account lacks proper credentials. So it seems like we've got a couple issues here: did we solve the SSL cert issue on the ADL side as you asked? And should I be using a different account to create new tenants on B2? If not, then I need help to grant my username the proper access. Thanks!

KatharineV commented 1 year ago

Hi team! I'd like to mark this ticket as the next top priority, after full text search, the Universal Viewer, and CSV/OAI import bugs are resolved.

ShanaLMoore commented 1 year ago

https://assaydepot.slack.com/archives/G0311DN2YCA/p1627684794017500

https://scientist-inc.sentry.io/issues/4030756171/?project=6745020&query=is%3Aunresolved&referrer=issue-stream

ShanaLMoore commented 1 year ago

When attempting to create a new tenant, it fails with a 401 unauthorized message.

We discovered that the new tenant had the wrong solr admin password. And when we edit the new tenant we could see the the solr_endpoint value was wrong.

April updated the solr admin value in rancher.

Additionally April says we have a DNS/Ingress issue that she will take on in her next maintenance sprint.

We also need to take a look at production. it is having similar issues.

ShanaLMoore commented 1 year ago

Marked High Priority

NOTE: Katharine's desire is to have this completed by her board meeting (April 11th).

ShanaLMoore commented 1 year ago

After deploying, we tried to make a tenant called 'demo'. We still receive a 500 error:

Image

The logs aren't very descriptive though and we aren't seeing anything from sentry. 

I, [2023-04-06T23:06:56.789464 scientist-softserv/adventist-dl#1] INFO -- : [b8f27878a026453827efda0689e83886] Completed 500 Internal Server Error in 3857ms (ActiveRecord: 2426.5ms)

hyku-dev-hyrax-68778bbfff-mv85n_hyrax.log

cc @orangewolf from this PR, are you saying that we need to add each tenant url to the env - tmpl.yml files BEFORE we attempt to create them on staging??

ShanaLMoore commented 1 year ago

removing the need rework label. I misunderstood what Rob meant by the description of his PR.

Note you will still need to add the tenant URL to the ingress section in ops/staging-deploy.tmpl.yaml or ops/production-deploy.tmpl.yaml as we do not have the ability to issue wildcard certs. the actual name (acme.b2.adventistdigitallibrary.org for example) has to be in the ops/production-deploy.tmpl.yaml and then deployed for it to work.

It sounds like if we want a new tenant there will need to be a process vs creating them on the fly. This is due to the the way their dns is done.

The process is adding their urls to the ingress section of the env's tmp.yml files. Deploying to the env, then creating the tenants.

I (or any dev) will test this by adding a demo tenant to staging. Diem will not be able to QA this via the UI until someone has added the url to the ingress sections.

kirkkwang commented 1 year ago

After deploy I still cannot create a new tenant, moving this ticket back for now

ShanaLMoore commented 1 year ago

I think I just realized why this is failing. I am going to retest and report back. For adventist, we should've made the change in dev-deploy.tmpl.yaml, not staging-deploy.tmpl.yaml.

BEFORE:

Image

UPDATE: it looks like this was tested correctly after all

creating a tenant still causes error:

I deployed the following PR to staging. It should have allowed me to create a qa tenant.

ref: https://github.com/scientist-softserv/adventist-dl/pull/479

Image

orangewolf commented 1 year ago

in the end there were two issues here. 1 a setting missing on Adventist Dev Template. and 2 a complex dance to get wildcard dns set up and ssl certs issued.

aprilrieger commented 1 year ago

Testing this ticket for the DNS issue we were trying to resolve on TT6. I see that I am able to create a tenant. I am able to navigate to the tenant and see it is active without needing to add any additional hosts to the ingress. I see a cert that covers the tenant that is active.

However, as a proprietor, when I create a new tenant I do receive a rails error page after adding a tenant short name and clicking save.

When trying to create a collection or work or anything in that tenant, you get errors. After navigating to the manage tenant page form the proprietor account you can see that the solr url is not correct: http://127.0.0.1:8983/solr/b4d4ff82-f92b-48a4-9b95-2d074e0fb2d9 the http://127.0.0.1:8983/ part should be pointed to the clusters solr url.

When I go to the production instance I see the env variables are correct for SETTINGSSOLRURL && SOLR_URL are set correctly.

Here are the sentry error log captured with the error: https://scientist-inc.sentry.io/issues/3984997772/?project=6745020&query=is%3Aunresolved&referrer=issue-stream&stream_index=0

Image Image Image Image

aprilrieger commented 1 year ago

Testing this ticket after deploying again made it all work. This is ready for client testing.

KatharineV commented 1 year ago

I tested on staging and created a new tenant, which I can view: https://wau.s2.adventistdigitallibrary.org/

It works! Thank you.