Deploy new platform in parallel

canasdiaz commented 5 years ago

The aim of this ticket is to create a parallel platform to allow Mozilla staff creating a report.

canasdiaz commented 5 years ago

This is the status:

Two nodes ready to deploy UI and data collection/enrichment software
SQL server is ready, data needs to be imported
ES server is ready, data is being imported from production node

Next steps:

when ES is ready, the UI will be deployed
Hastall will be deployed after the Kibiter UI
last step will be the data enrichment software

According to the current speed, platform should be fully available on Monday.

canasdiaz commented 5 years ago

deployment of UI is blocked by proxy credentials. Waiting ..

canasdiaz commented 5 years ago

This is a brief summary of the status:

ES data is ready
SQL data is being imported
UI is blocked by the domain sotrar.mozilla.community which is not available

Next steps:

when the domain is available the deployment of Kibiter will be finished
Hastall will be deployed after the Kibiter UI
last step will be the data enrichment software

According to the current speed, platform should be fully available on Monday.

This is the complete log:

We got the proxy credentials
The domain sotrar.mozilla.community is not available from internet yet and it is blocking us to test the UI deployment

➜  ~ dig sotrar.mozilla.community 

; <<>> DiG 9.11.5-P4-1-Debian <<>> sotrar.mozilla.community
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 53156
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;sotrar.mozilla.community.      IN      A

;; AUTHORITY SECTION:
mozilla.community.      899     IN      SOA     ns-1787.awsdns-31.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 708 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: vie mar 08 07:19:31 CET 2019
;; MSG SIZE  rcvd: 140

hmitsch commented 5 years ago

Looks like the DNS propagation is done now:

$ dig sotrar.mozilla.community

; <<>> DiG 9.10.6 <<>> sotrar.mozilla.community
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54830
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;sotrar.mozilla.community.  IN  A

;; ANSWER SECTION:
sotrar.mozilla.community. 3600  IN  A   62.138.239.45
sotrar.mozilla.community. 3600  IN  A   62.138.238.45

;; Query time: 334 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Sat Mar 09 00:12:01 CET 2019
;; MSG SIZE  rcvd: 85

canasdiaz commented 5 years ago

I'm afraid DNS is not ready yet (the DNS propagation may take several hours)

imagen

source: https://dns.google.com/query?name=sotrar.mozilla.community

(the result should look similar to this)

canasdiaz commented 5 years ago

This is a brief summary of the status:

data is imported from production platform
UI is blocked by the domain sotrar.mozilla.community which is not available (see my comment above)

Next steps:

when the domain is available the deployment of Kibiter will be finished (this should be quick)
Hastall (already deployed) will be tested after the Kibiter UI
last step will be the data enrichment software

According to the current speed, platform should be fully available as soon as the domain is working (hopefully on Monday)

canasdiaz commented 5 years ago

This is a brief summary of the status:

data is imported from production platform
Kibiter is available at sotrar.mozilla.community
ES is also available
Data analysis software is deployed. A new version of the Git index is being produced

Next steps:

Hastall

canasdiaz commented 5 years ago

All components are deployed but Hatstall is 100% ready. This must be fixed before closing this ticket . Currently the search box is not working, when u try to search something you get an error saying the connection was restarted. This could be due to the long time the SQL queries need (more than 15 secs).

Due to the proxy is not the same one we have in production we need to change some parameters for this.

canasdiaz commented 5 years ago

This is a brief summary of the status:

Hastall search box is not working
The rest is ready

I've been doing some tests with a smaller SQL database and it works successfully. So, the issue is isolated in the proxy which is restarting the connection 10 secs after the request is sent. The SQL engine does not reply before that time due to the size of the database (around 200K identities). So, the thing to be changed is the timeout in the proxy we deployed. I've just asked @johngian for help so when he has a slot we'll be able to finish this. When this is fixed I'll apply the same setup to analytics.m.c to make that instance more secure as requested in #187

CC @hmitsch

canasdiaz commented 5 years ago

Hatstall is finally working @hmitsch :dancer:

After applying a couple of modifications suggested by @johngian we have Hatstall protected by the AIM proxy. The only con of this new deployment is the first time you access to sotrar.m.c/* it takes around 20 secs to load everything. In any case I guess this is enough for the sotrar site.

Please check and close if you agree with me. I'm moving this to Done.

https://sotrar.mozilla.community/identities

canasdiaz commented 5 years ago

Can we close this ticket @hmitsch ?

mozilla / participation-metrics-org

Deploy new platform in parallel #191