search

publiclab / image-sequencer-app

An Image Processing server based on image-sequencer
GNU General Public License v3.0
7 stars 5 forks source link

Load testing! #19

Open jywarren opened 5 years ago

jywarren commented 5 years ago

Now, building on the list from https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23, let's see how big images/maps it can handle!

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json&scale=30 is real fast. seconds.

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/ceres--2/warpables.json&scale=30 -- It worked!!!! 25mb output in <2 minutes! Too big even to upload to GitHub! https://www.dropbox.com/s/m6v131cyynlwx0a/ceres--2.png?dl=0

(note that scale isn't working now -- #14 )

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 next:

Then:

tech4GT commented 5 years ago

Cool! I’m heading out now but I’ll get on this in a couple of hours.

jywarren commented 5 years ago

Great! Same!!

jywarren commented 5 years ago

Yes this link as I had above is correct for logs:

https://console.cloud.google.com/logs/viewer?resource=k8s_container%2Fcluster_name%2Fmapknitter%2Fnamespace_name%2Fdefault%2Fcontainer_name%2Fimage-sequencer-app&logName=projects%2Fpublic-lab%2Flogs%2Fstderr&expandAll=false&timestamp=2019-06-10T21:58:41.154000000Z&project=public-lab&authuser=1&minLogLevel=0&customFacets=&limitCustomFacetWidth=true&interval=JUMP_TO_TIME&scrollTimestamp=2019-06-10T21:57:32.444075029Z

On Sat, Jun 15, 2019, 9:32 AM Varun Gupta notifications@github.com wrote:

Cool! I’m heading out now but I’ll get on this in a couple of hours.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J26Z7UUHSCMIJS3CQ3P2TVNDA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYYIGI#issuecomment-502367257, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6JZ7RGRJJLC4ZWF4B4DP2TVNDANCNFSM4HWZEIJA .

jywarren commented 5 years ago

Ok actually there are no logs i can find since the 10th, but there may have been one more error then:

https://gist.github.com/jywarren/e46994aa9dec8d774245822365fb446f

But I don't know why we aren't seeing any other logs! Do we need this express app to generate more logging, at least for requests?

jywarren commented 5 years ago

Hmm, actually I'm seeing a This site can’t be reached / 34.74.118.242 refused to connect. error... for this URL (after initial redirect): click here

Ah, but a few moments later it did load, and generated a good export!

tech4GT commented 5 years ago

Hmm, Maybe this is a browser issue, the browser might not be waiting enough time for the server to process the response! Let me integrate the bucket with our code!

tech4GT commented 5 years ago

Then we would have a better understanding as to what is going on!

jywarren commented 5 years ago

Anyhow here are some maps with <10 images;

tech4GT commented 5 years ago

Are these working?

jywarren commented 5 years ago

I'm trying them one at a time... starting with the last

jywarren commented 5 years ago

ah for marshburg i get the same "refused to connect" error... but i'll wait a bit and see it it'll load later like the other did.

Update: Chrome keeps trying to reload that page, so we'll see...

tech4GT commented 5 years ago

Okay! I'll look into the cloud stuff meanwhile!

jywarren commented 5 years ago

Ah ok so it's showing in the cloud logs now! 

{
 insertId:  "f5rcvu9l8f5t9h1go"  
 labels: {…}  
 logName:  "projects/public-lab/logs/stderr"  
 receiveTimestamp:  "2019-06-15T14:40:45.103722159Z"  
 resource: {…}  
 severity:  "ERROR"  
 textPayload:  "(node:21) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added. Use emitter.setMaxListeners() to increase limit
"  
 timestamp:  "2019-06-15T14:40:39.359077955Z"  
}
tech4GT commented 5 years ago

Okay! Let me set the max listeners then!

jywarren commented 5 years ago

I thought you did already in https://github.com/publiclab/image-sequencer-app/commit/9aa949d79ce7d214785919705afd416bc64a82d3 ?

The marshburg map finally loaded, but after spinning for a few seconds, i now see This page isn’t working 34.74.118.242 didn’t send any data.

jywarren commented 5 years ago

Ah but in the logs i now see that it rebooted! Maybe it crashed. Maybe it has the maxListeners code loaded now...?

tech4GT commented 5 years ago

I did but maybe I need to set it for every request separately!

tech4GT commented 5 years ago

Oh!

jywarren commented 5 years ago

Yeah i'm still getting the MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added. Use emitter.setMaxL... error in the logs. It's quite helpful to run it and watch the logs live ! Try it, you should have permissions...

jywarren commented 5 years ago

You can watch new versions build here, though they take a little longer to actually get to the cluster, i think: https://console.cloud.google.com/cloud-build/builds?authuser=1&project=public-lab

tech4GT commented 5 years ago

Awesome! How do I deploy the code which contains my credentials for uploading?

tech4GT commented 5 years ago

I mean I shouldn't just push this to github, right?

jywarren commented 5 years ago

yeah... don't worry about cloud storage just yet! Maybe run it locally and use an environment variable if you can? Then we'll add the creds safely.

tech4GT commented 5 years ago

Okay, but how would do test it out then. I have a strong feeling that the browser might be sending repeat requests after every 30 seconds or so, which causes the cluster to crash. Maybe I'll try to request the URL with curl?

tech4GT commented 5 years ago

Okay, I'm getting connection refused error! Why is it crashing like this?

tech4GT commented 5 years ago

Okay I don't see the listeners exceeded log anymore, but for some reason I am not getting the correct response from the server.

tech4GT commented 5 years ago

@jywarren Are you also facing the same thing on your end?

jywarren commented 5 years ago

I'm not at a computer right now, sorry!!!

On Sat, Jun 15, 2019, 11:51 AM Varun Gupta notifications@github.com wrote:

@jywarren https://github.com/jywarren Are you also facing the same thing on your end?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J7LJAVNGZIN2KWERGTP2UFXDA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY2X6Q#issuecomment-502377466, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J5DGH5T53X2YNV53SDP2UFXDANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

Okay cool! I am working on the cloud stuff right now! Will circle back to this after that is working!

tech4GT commented 5 years ago

Okay @jywarren I have written the status endpoint code locally! Now how do we deploy it? Should I just gitignore my secrets and push it here, but that would break the current production build?

jywarren commented 5 years ago

Just enable it with a flag maybe, or can you check presence of env variables and only upload if they exist?

On Sat, Jun 15, 2019, 12:54 PM Varun Gupta notifications@github.com wrote:

Okay @jywarren https://github.com/jywarren I have written the status endpoint code locally! Now how do we deploy it? Should I just gitignore my secrets and push it here, but that would break the current production build?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6JZZL25WZBDPHL26ZADP2UNDFA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY35MQ#issuecomment-502382258, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6JYVBW6HVTEQ4BCPLRDP2UNDFANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

I'm not really sure how to do that, maybe I can push the new commits in, once someone configures the cluster?

tech4GT commented 5 years ago

For now I'll move on to why it's crashing. I'm not even able to get a response for small maps like the ceres--2

tech4GT commented 5 years ago

Also i tried to redirect requests from the user, from the status endpoint to the file in the bucket once the export is done. But I don't think it's publicly accessible, we may want to change that.

jywarren commented 5 years ago

You may need to make each file public.

Can we make cloud storage depend on a get parameter like &useCloudStorage=true ?

On Sat, Jun 15, 2019, 1:02 PM Varun Gupta notifications@github.com wrote:

Also i tried to redirect requests from the user, from the status endpoint to the file in the bucket once the export is done. But I don't think it's publicly accessible, we may want to change that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J2LD4MQBH23Y5QRAPDP2UOA5A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY4CPI#issuecomment-502382909, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J32AWAGXPFXN67KGV3P2UOA5ANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

Okay I was researching memory leaks and stack overflow says that even extensive logs can leak memory by tying down the event loop. Disabling the progress bars might just help!!

tech4GT commented 5 years ago

You may need to make each file public. Can we make cloud storage depend on a get parameter like &useCloudStorage=true ? … On Sat, Jun 15, 2019, 1:02 PM Varun Gupta @.***> wrote: Also i tried to redirect requests from the user, from the status endpoint to the file in the bucket once the export is done. But I don't think it's publicly accessible, we may want to change that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19?email_source=notifications&email_token=AAAF6J2LD4MQBH23Y5QRAPDP2UOA5A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY4CPI#issuecomment-502382909>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J32AWAGXPFXN67KGV3P2UOA5ANCNFSM4HWZEIJA .

Sure! We already have an upload option! I'll use it good idea! Also is there some extra parameter while creating the file which makes it public?

jywarren commented 5 years ago

Yes it's in the Google documentation!

On Sat, Jun 15, 2019, 1:09 PM Varun Gupta notifications@github.com wrote:

You may need to make each file public. Can we make cloud storage depend on a get parameter like &useCloudStorage=true ? … <#m8720807423403805511> On Sat, Jun 15, 2019, 1:02 PM Varun Gupta @.***> wrote: Also i tried to redirect requests from the user, from the status endpoint to the file in the bucket once the export is done. But I don't think it's publicly accessible, we may want to change that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J2LD4MQBH23Y5QRAPDP2UOA5A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY4CPI#issuecomment-502382909>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J32AWAGXPFXN67KGV3P2UOA5ANCNFSM4HWZEIJA .

Sure! We already have an upload option! I'll use it good idea! Also is there some extra parameter while creating the file which makes it public?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J42VTJRTQFSBCGS7DDP2UO2XA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY4GGY#issuecomment-502383387, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6JYDNHY5C3MSFCBLB33P2UO2XANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

Okay pushed it in! Now we can enable it using upload=true in the query.

tech4GT commented 5 years ago

Working on making it public now!

tech4GT commented 5 years ago

Okay I put the redirect in place!! Now http://localhost:4000/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json&upload=true should give an immediate response and further, http://localhost:4000/api/v2/status/?pid=47516 is downloading the file. Right now It's embedded in an html page but we can change that.

tech4GT commented 5 years ago

@jywarren Please publish the new version of image sequencer onto npm whenever you can find the time, let's see if the pace progress bars are at fault here.

tech4GT commented 5 years ago

Okay I pushed in the new code, but it is not yet deployed! @jywarren any idea why?

jywarren commented 5 years ago

Hi @tech4GT can you update the status here?

Varun Gupta @tech4GT Jun 18 21:28 - Yeah! The code which changes max listeners is now deployed! Let’s see if we still get the same error.

What should we test? This?

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json&upload=true

returns:

Your Image is exporting, to load Image please visit, http://34.74.118.242/api/v2/status/?pid=32

http://34.74.118.242/api/v2/status/?pid=32 then seems to hang!

jywarren commented 5 years ago

Is it possible the requests aren't going to the same container instance? Do we need to use an identifier (unlike pid) that would be unique across the cluster?

jywarren commented 5 years ago

Many commits were made -- https://github.com/publiclab/image-sequencer-app/commit/046bbf75cd85394646e76110a244af3e5e633c27

Maybe it was this one for @Divy123's questions?

https://github.com/publiclab/image-sequencer-app/commit/1256685ebe4e81e70cad5ad9ba1ffc70a8ebb69d

Divy123 commented 5 years ago

Ya that is something that can be done, but I don't know where this code should lie in IS.

tech4GT commented 5 years ago

@jywarren the upload option won’t work coz we don’t have credentials on the cluster as of now. Let’s test it without that option and see if we get the same error in the logs!

jywarren commented 5 years ago

OK, testing: http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json

OK it almost instantly worked... now something bigger:

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/ceres--2/warpables.json

That worked. http://34.74.118.242/api/v2/export/?url=https://mapknitter.org/maps/marshburg/warpables.json next...

jywarren commented 5 years ago

http://34.74.118.242/api/v2/export/?url=https://mapknitter.org/maps/marshburg/warpables.json stalled the first time, so i restarted it...

http://34.74.118.242/api/v2/process/?upload=undefined&steps=[%7B%22id%22:1,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314640/DJI_0077.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1099%252C1897%7Cne%3A2670%252C1772%7Cse%3A2843%252C2231%7Csw%3A1314%252C2357%7D%22%7D,%7B%22id%22:2,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314641/DJI_0074.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1536%252C2272%7Cne%3A2287%252C2242%7Cse%3A2346%252C2495%7Csw%3A1542%252C2508%7D%22%7D,%7B%22id%22:3,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314642/DJI_0076.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A955%252C1927%7Cne%3A2877%252C1812%7Cse%3A3055%252C2347%7Csw%3A1095%252C2440%7D%22%7D,%7B%22id%22:4,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314643/DJI_0078.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1225%252C1803%7Cne%3A2581%252C1697%7Cse%3A2732%252C2104%7Csw%3A1375%252C2210%7D%22%7D,%7B%22id%22:5,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314644/DJI_0075.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1440%252C2138%7Cne%3A2344%252C2100%7Cse%3A2444%252C2423%7Csw%3A1513%252C2465%7D%22%7D,%7B%22id%22:6,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314646/DJI_0081.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A715%252C1413%7Cne%3A2958%252C1151%7Cse%3A3197%252C1888%7Csw%3A1140%252C2096%7D%22%7D,%7B%22id%22:7,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314654/DJI_0088.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1231%252C707%7Cne%3A2498%252C565%7Cse%3A2686%252C993%7Csw%3A1378%252C1126%7D%22%7D,%7B%22id%22:8,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314655/DJI_0090.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1992%252C230%7Cne%3A3594%252C98%7Cse%3A3762%252C589%7Csw%3A2161%252C720%7D%22%7D,%7B%22id%22:9,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314658/DJI_0093.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1078%252C148%7Cne%3A2596%252C0%7Cse%3A2752%252C470%7Csw%3A1234%252C618%7D%22%7D,%7B%22id%22:10,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314671/DJI_0035.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A0%252C1781%7Cne%3A1270%252C1502%7Cse%3A1620%252C1850%7Csw%3A385%252C2141%7D%22%7D,%7B%22id%22:11,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/314689/DJI_0055.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1735%252C755%7Cne%3A994%252C755%7Cse%3A994%252C525%7Csw%3A1735%252C525%7D%22%7D,%7B%22id%22:12,%22input%22:1,%22depends%22:[1,2,3,4,5,6,7,8,9,10,11],%22steps%22:%22canvas-resize%7Bwidth:7524%7Cheight:5016%7Cx:1099%7Cy:1772%7D,import-image%7Burl:output%3E2%7D,overlay%7Bx:1536%7Cy:2242%7D,import-image%7Burl:output%3E3%7D,overlay%7Bx:955%7Cy:1812%7D,import-image%7Burl:output%3E4%7D,overlay%7Bx:1225%7Cy:1697%7D,import-image%7Burl:output%3E5%7D,overlay%7Bx:1440%7Cy:2100%7D,import-image%7Burl:output%3E6%7D,overlay%7Bx:715%7Cy:1151%7D,import-image%7Burl:output%3E7%7D,overlay%7Bx:1231%7Cy:565%7D,import-image%7Burl:output%3E8%7D,overlay%7Bx:1992%7Cy:98%7D,import-image%7Burl:output%3E9%7D,overlay%7Bx:1078%7Cy:0%7D,import-image%7Burl:output%3E10%7D,overlay%7Bx:0%7Cy:1502%7D,import-image%7Burl:output%3E11%7D,overlay%7Bx:994%7Cy:525%7D%22%7D]