Set up the production environment of the TDC data portal

Uchechukwu-Onye-Igbo commented 4 weeks ago

When the TDC portal is ready for deployment, I want to set up a reliable, scalable production environment, so that users can access and interact with the portal seamlessly with optimal performance and security.

Tasklist (TBD - updated by the developer)

[x] DB setup
[x] solr setup
[x] redis setup
[x] dns setuo
[x] frontend deployment
[x] ckan deployment
[x] datasets migration
[x] CKAN CNAME or A record pointing to the Datopian's load balancer. - CLIENT task
[x] FRONTEND CNAME or A record pointing to the Datopian's load balancer. - CLIENT task
[x] Once they setup CNAME, we will update URL's from our side

Acceptance Criteria

Given that I am a user public user, I can

[x] access the TDC portal via a secure public link
[x] I can log in and perform all functions that the portal has been designed to do effortlessly
[x] I can securely link my GitHub account to create an account on the portal without security frailties

CLIENT TASK

In order to setup custom domain

The client needs to add a CNAME that points to the datopian domain of that instance, for example PROD instance is ckan.tdc.prod.datopian.com OR
If the client doesn't want to add the CNAME to internal URL for some reason, they can add the A-record pointing to the nginx loadbalancer.
- For example A record data.domain.org pointing to nginx-prod-lb IP i.e 35.205.166.46.

khaeru commented 3 weeks ago

The client needs to add a CNAME that points to the datopian domain of that instance, for example PROD instance is ckan.tdc.prod.datopian.com

Done:

$ dig ckan.transport-data.org

; <<>> DiG 9.20.0-2ubuntu3-Ubuntu <<>> ckan.transport-data.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49370
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;ckan.transport-data.org.       IN      A

;; ANSWER SECTION:
ckan.transport-data.org. 14400  IN      CNAME   ckan.tdc.prod.datopian.com.

;; AUTHORITY SECTION:
datopian.com.           600     IN      SOA     glen.ns.cloudflare.com. dns.cloudflare.com. 2355853271 10000 2400 604800 1800

;; Query time: 78 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Thu Oct 31 11:42:45 CET 2024
;; MSG SIZE  rcvd: 151

I also gave @nicolas-becker manager permissions for this domain, so that he can make other further changes as required.

Mikanebu commented 1 week ago

Dear @khaeru ! We configured CKAN site to ckan.transport-data.org. It is live now. Second task will be setting up same configuration for FRONTEND site. You need to create similar CNAME for https://frontend.tdc.prod.datopian.com/. https://ckan.tdc.prod.datopian.com which is now ckan.transport-data.org will serve as backend/admin portal, not for users. cc @Uchechukwu-Onye-Igbo @nicolas-becker.

nicolas-becker commented 1 week ago

Dear @Mikanebu, here's the CNAME for the frontend. Let me know if you need any further settings. Thanks!

> Resolve-DnsName portal.transport-data.org

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
portal.transport-data.org      CNAME  14400 Answer     frontend.tdc.prod.datopian.com

Name       : nginx-prod-lb.datopian.com
QueryType  : A
TTL        : 300
Section    : Answer
IP4Address : 35.205.166.46

Name                   : datopian.com
QueryType              : SOA
TTL                    : 1800
Section                : Authority
NameAdministrator      : dns.cloudflare.com
SerialNumber           : 2357447982
TimeToZoneRefresh      : 10000
TimeToZoneFailureRetry : 2400
TimeToExpiration       : 604800
DefaultTTL             : 1800

cc: @Uchechukwu-Onye-Igbo @khaeru

Mikanebu commented 1 week ago

@nicolas-becker Frontend portal is configured from our side. It is up and running https://portal.transport-data.org/. cc @Uchechukwu-Onye-Igbo @khaeru

nicolas-becker commented 5 days ago

Thanks @Mikanebu . I am currently experiencing the issue that the landing page of the frontend portal does not finish loading. Is that possibly related to some work you are doing?

I also flagged this error to @demenech when I tried to add a new organization:

This happened when requesting a new Org during onboarding and also happens when doing the same via the Dashboard.

On a minor note,

I am still seing a Test Org among the organizations, it would be great if you could drop it from the dev environment.
Would it be possible to transfer the Topics I had created in the dev instance?

And one last general question: How do we run updates? I.e. when we do modifications via the Github DMS, will those changes be reflected only on the dev instance or on both? The other day I noted that there was a Test Person appearing in the People section of the About Us page, so I deleted the file in the DMS. The profile was deleted on both environments

demenech commented 5 days ago

hello @nicolas-becker

The email issue should be resolved now

I also fixed an issue with some images not loading due to a misconfiguration.

"Test Org" deleted.

Re transferring topics, can you confirm this is the list you want to transfer, please?

["aviation", "logistics", "my-child-topic", "public-transportation", "qa-topic-oct-4", "rail", "transport-emissions", "vehicles", "water-transportation"]

Regarding your last question, @willy1989cv can you take a look please?

nicolas-becker commented 4 days ago

Re transferring topics, can you confirm this is the list you want to transfer, please?

It should be the following 7 topics:

["aviation", "logistics", "public-transportation", "rail", "transport-emissions", "vehicles", "water-transportation"]

Thanks @demenech

willy1989cv commented 4 days ago

And one last general question: How do we run updates? I.e. when we do modifications via the Github DMS, will those changes be reflected only on the dev instance or on both? The other day I noted that there was a Test Person appearing in the People section of the About Us page, so I deleted the file in the DMS. The profile was deleted on both environments

Yes, @nicolas-becker they are both pointed to the same "main" branch.

demenech commented 4 days ago

hello @nicolas-becker

The file loading issue (e.g. images) on the FE application has been fixed. Please validate. Thanks!

CC: @osahon-okungbowa @Uchechukwu-Onye-Igbo

nicolas-becker commented 1 day ago

And one last general question: How do we run updates? I.e. when we do modifications via the Github DMS, will those changes be reflected only on the dev instance or on both? The other day I noted that there was a Test Person appearing in the People section of the About Us page, so I deleted the file in the DMS. The profile was deleted on both environments

Yes, @nicolas-becker they are both pointed to the same "main" branch.

Okay, so this means, that all changes in the GitHub DMS are directly pushed to both environments. The difference between the two instances is the connected CKAN backend then? @willy1989cv

nicolas-becker commented 1 day ago

The file loading issue (e.g. images) on the FE application has been fixed. Please validate. Thanks!

Hey @demenech , sorry I am not entirely sure which issue you are referring to here... Do you mean the issue trying to request new orgs?

demenech commented 1 day ago

Hi @nicolas-becker

Refering to, for example, the loading indicator not disappearing.

willy1989cv commented 23 hours ago

Okay, so this means, that all changes in the GitHub DMS are directly pushed to both environments. The difference between the two instances is the connected CKAN backend then? @willy1989cv

yes @nicolas-becker, exaclty.

nicolas-becker commented 13 hours ago

Hi @demenech

"Test Org" deleted

Unfortunately I am still seeing the Test Org in the dataset browser page:

Also, would it be possible to grant sysadmin rights to my user in the productive portal please?

transport-data / tdc-data-portal