search

alkem-io / infrastructure

Creates deployments, managing infrastructure scripts, devops, monitoring, ...
European Union Public License 1.2
4 stars 0 forks source link

Connection to cluster is unstable #30

Closed techsmyth closed 3 years ago

techsmyth commented 3 years ago

Describe the bug Submitting up sample data via the graphql api is giving sporadic but frequest connection reset issues.

To Reproduce Steps to reproduce the behavior:

  1. Ensure have demo-18 branch from Demo repository - it has better logging so easier to see what is happening.
  2. Reset the db on the cluster (npm run test-db-reset directly on the pod)
  3. Submit the full set of sample data from Demo + the challenges from Odyssey to the server
  4. Look at how many of the challenge (larger) submissions fail due to connection resets. From my first run below in total 3 out of 15 worked.
  5. Repeat above steps - get a different set of request failures, with the second time 8 out of 15 succeeding.

I also saw this behaviour yesterday. Both trails are below.

It seems the smaller requests are fine, but a challenge full upload is having some sporadic issues.

Note that running the above steps locally works for me 100% reliably.

Expected behavior All requests should reliably complete.

Run 1 neil@Spectre360v2 MINGW64 ~/Documents/Dev_CT/Demo (demo-18) $ npm run sample-data

cherrytwist-demo@0.1.0 sample-data C:\Users\neil\Documents\Dev_CT\Demo ts-node-dev src/submit-sample-data.ts

ts-node-dev ver. 1.0.0-pre.60 (using ts-node ver. 8.10.2, typescript ver. 4.0.2) Connected to ecoverse: CherryTwist dogfood Loading users using: ./src/queries/create-user ........user(1) submitted from: create-user-ninja3.json ......................accepted: SuperNinja3, with id: 9 ........user(2) submitted from: create-user-ninja4.json ......................accepted: SuperNinja4, with id: 10 Loading of 2 user(s) completed (out of 2) Loading userGroups using: ./src/queries/create-userGroup ........userGroup(1) submitted from: create-userGroup-ninjas.json ......................accepted: Teenage Mutant Ninja Turtles, with id: 8 Loading of 1 userGroup(s) completed (out of 1) Loading organisations using: ./src/queries/create-organisation ........organisation(1) submitted from: create-organisation-ct.json ......................accepted: Cherrytwist, with id: 2 Loading of 1 organisation(s) completed (out of 1) Loading challenges using: ./src/queries/create-challenge ........challenge(1) submitted from: balance-the-grid.json Submission to the server from balance-the-grid.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(2) submitted from: compliant-air-cargo.json ......................accepted: Compliant air cargo flow, with id: 5 ........challenge(3) submitted from: conflict-prevention.json Submission to the server from conflict-prevention.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(4) submitted from: decarbonization.json Submission to the server from decarbonization.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(5) submitted from: driving-the-adoption.json Submission to the server from driving-the-adoption.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(6) submitted from: energy-singularity.json ......................accepted: Interact with digital, with id: 6 ........challenge(7) submitted from: freight-packaging.json ......................accepted: Circular economy for freight packaging, with id: 7 ........challenge(8) submitted from: inclusive-safety.json Submission to the server from inclusive-safety.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(9) submitted from: kadaster-buildings-personality.json Submission to the server from kadaster-buildings-personality.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(10) submitted from: ll-scalable-and-inclusive.json Submission to the server from ll-scalable-and-inclusive.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(11) submitted from: marine-biodiversity.json Submission to the server from marine-biodiversity.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(12) submitted from: open-source-911.json Submission to the server from open-source-911.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(13) submitted from: plastic-pollution.json Submission to the server from plastic-pollution.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(14) submitted from: smart-meter-data.json Submission to the server from smart-meter-data.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(15) submitted from: ssi-in-action.json Submission to the server from ssi-in-action.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET Loading of 3 challenge(s) completed (out of 15) Loading data complete!

Run 2 neil@Spectre360v2 MINGW64 ~/Documents/Dev_CT/Demo (demo-18) $ npm run sample-data

cherrytwist-demo@0.1.0 sample-data C:\Users\neil\Documents\Dev_CT\Demo ts-node-dev src/submit-sample-data.ts

ts-node-dev ver. 1.0.0-pre.60 (using ts-node ver. 8.10.2, typescript ver. 4.0.2) Connected to ecoverse: CherryTwist dogfood Loading users using: ./src/queries/create-user ........user(1) submitted from: create-user-ninja3.json ......................accepted: SuperNinja3, with id: 9 ........user(2) submitted from: create-user-ninja4.json ......................accepted: SuperNinja4, with id: 10 Loading of 2 user(s) completed (out of 2) Loading userGroups using: ./src/queries/create-userGroup ........userGroup(1) submitted from: create-userGroup-ninjas.json ......................accepted: Teenage Mutant Ninja Turtles, with id: 8 Loading of 1 userGroup(s) completed (out of 1) Loading organisations using: ./src/queries/create-organisation
........organisation(1) submitted from: create-organisation-ct.json ......................accepted: Cherrytwist, with id: 2 Loading of 1 organisation(s) completed (out of 1) Loading challenges using: ./src/queries/create-challenge
........challenge(1) submitted from: balance-the-grid.json ......................accepted: Balance the grid, with id: 5 ........challenge(2) submitted from: compliant-air-cargo.json Submission to the server from compliant-air-cargo.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(3) submitted from: conflict-prevention.json Submission to the server from conflict-prevention.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(4) submitted from: decarbonization.json ......................accepted: Consumer- and organization driven decarbonization, with id: 6 ........challenge(5) submitted from: driving-the-adoption.json Submission to the server from driving-the-adoption.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(6) submitted from: energy-singularity.json ......................accepted: Interact with digital, with id: 7 ........challenge(7) submitted from: freight-packaging.json Submission to the server from freight-packaging.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(8) submitted from: inclusive-safety.json ......................accepted: Inclusive safety communities, with id: 8 ........challenge(9) submitted from: kadaster-buildings-personality.json ......................accepted: Give buildings a personality, with id: 9 ........challenge(10) submitted from: ll-scalable-and-inclusive.json Submission to the server from ll-scalable-and-inclusive.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(11) submitted from: marine-biodiversity.json ......................accepted: Protecting marine biodiversity, with id: 10 ........challenge(12) submitted from: open-source-911.json ......................accepted: Open-source 911 for digital space, with id: 11 ........challenge(13) submitted from: plastic-pollution.json Submission to the server from plastic-pollution.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(14) submitted from: smart-meter-data.json Submission to the server from smart-meter-data.json rejected: FetchError: request to http://dev.cherrytwist.org/graphql failed, reason: read ECONNRESET ........challenge(15) submitted from: ssi-in-action.json ......................accepted: Self-sovereign identity in action, with id: 12 Loading of 8 challenge(s) completed (out of 15) Loading data complete!

techsmyth commented 3 years ago

Works 100% with https for Neil. Still problems with http. Works also with http for Aleks.