search

publiclab / mapknitter-exporter-sinatra

A minimal Sinatra app to run MapKnitter exports in the cloud
GNU General Public License v3.0
3 stars 4 forks source link

Load testing!!! #23

Open jywarren opened 5 years ago

jywarren commented 5 years ago

We'll find a bigger one too!!!

jywarren commented 5 years ago

https://mapknitter.org/profile/eustatic and https://mapknitter.org/profile/cfastie probably have some GIANT ones.

image

icarito commented 5 years ago

Please point export.mapknitter.org to this IP: 104.196.68.120 It should work with large images now! :crossed_fingers:

icarito commented 5 years ago

imagen Cluster is churning away! :dark_sunglasses:

icarito commented 5 years ago

:steam_locomotive:

jywarren commented 5 years ago

Ok did this although last time there were 4 entries, now only one?

Thanks!!!

On Sat, May 4, 2019, 2:39 AM Sebastian Silva notifications@github.com wrote:

🚂

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-489300176, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J5WGI6NRXZYDSGEP43PTUVPPANCNFSM4HKCHODA .

jywarren commented 5 years ago

Hmm, maybe I did it wrong? This site can’t be reached export.mapknitter.org took too long to respond.

I'll try directly at 104.196.68.120

jywarren commented 5 years ago

Hmm, the log isn't showing any requests reaching it... strange.

jywarren commented 5 years ago

Been trying https://104.196.68.120/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30 and https://export.mapknitter.org/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30

jywarren commented 5 years ago

aha!!! http. duh.

jywarren commented 5 years ago

http://export.mapknitter.org/export?url=https://mapknitter.org/maps/pvdtest/warpables.json&scale=30 works! Great.

Hmm. Despite my changes in #27 it's still only displaying a string, not a JSON object with the location of the JPG file: http://export.mapknitter.org/id/1556986755/status.json

jywarren commented 5 years ago

OK, outputting geotiff and zip, and echoing more of the status and paths to console: https://github.com/publiclab/mapknitter-exporter-sinatra/pull/28

jywarren commented 5 years ago

http://export.mapknitter.org/id/1556986755/status.json seems still stuck on generating jpg; maybe we'll learn more from the logs after #28

jywarren commented 5 years ago

Strange, i'm not seeing any of the kinds of log output we had seen previously, even last night, like:


2019-05-04T04:40:20.413119293Z- [-33400560.850078963, 11671074.30096948]
2019-05-04T04:40:20.413170584Zwarping 11 of 19
2019-05-04T04:40:20.573044610Zconvert -contrast-stretch 0 public/warps/193358-working/w193358-a_step_11_IMG_5313.JPG -crop 4287x4287+0+0! -flatten -distort Perspective '0,0 0,383 4287,0 99,0 4287,2821 314,122 0,2821 237,506' -flatten -crop 506x506+0+0! +repage public/warps/4621/w193358.png

Wonder why? I'm looking in the logs for Cloud Build... maybe that's no longer the right place...

jywarren commented 5 years ago

https://github.com/publiclab/mapknitter-exporter-sinatra/pull/28 merged, deploying revision once it's ready...

jywarren commented 5 years ago

Aha, i bet i'm in the wrong cloud system. Kubernetes maybe?

jywarren commented 5 years ago

Looks like it. I'll try deploying it again.

jywarren commented 5 years ago

OK, i'm a bit out of my depth now. But the latest main branch of this repo is ready to go!

icarito commented 5 years ago

I'm deploying from kubernetes

icarito commented 5 years ago

Here's some pointers to docs for kubernetes... I'm using the console for looking at logs sanely:

https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app

$ kubectl get pod NAME                                         READY   STATUS    RESTARTS   AGE mapknitter-exporter-kubes-657bc75dbf-hfdb8   1/1     Running   0          3m48s mapknitter-exporter-kubes-657bc75dbf-zs829   1/1     Running   0          3m50s mapknitter-exporter-kubes-657bc75dbf-zxjf7   1/1     Running   0          3m33s

$ kubectl log -f NAME

for log...

:-)

On 4/05/19 11:34, Jeffrey Warren wrote:

Strange, i'm not seeing any of the kinds of log output we had seen previously, even last night, like:

|2019-05-04T04:40:20.413119293Z- [-33400560.850078963, 11671074.30096948] 2019-05-04T04:40:20.413170584Zwarping 11 of 19 2019-05-04T04:40:20.573044610Zconvert -contrast-stretch 0 public/warps/193358-working/w193358-a_step_11_IMG_5313.JPG -crop 4287x4287+0+0! -flatten -distort Perspective '0,0 0,383 4287,0 99,0 4287,2821 314,122 0,2821 237,506' -flatten -crop 506x506+0+0! +repage public/warps/4621/w193358.png |

Wonder why?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-489342414, or mute the thread https://github.com/notifications/unsubscribe-auth/AABQYS3ADNC27VDW2BPHVQ3PTW3HZANCNFSM4HKCHODA.

jywarren commented 5 years ago

Thanks!!! It's strange, I don't see the other files uploading to the bucket. Hmm. But we're getting proper Json now!

On Sat, May 4, 2019, 3:03 PM Sebastian Silva notifications@github.com wrote:

Here's some pointers to docs for kubernetes... I'm using the console for looking at logs sanely:

https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app

$ kubectl get pod NAME READY STATUS RESTARTS AGE mapknitter-exporter-kubes-657bc75dbf-hfdb8 1/1 Running 0 3m48s mapknitter-exporter-kubes-657bc75dbf-zs829 1/1 Running 0 3m50s mapknitter-exporter-kubes-657bc75dbf-zxjf7 1/1 Running 0 3m33s

$ kubectl log -f NAME

for log...

:-)

On 4/05/19 11:34, Jeffrey Warren wrote:

Strange, i'm not seeing any of the kinds of log output we had seen previously, even last night, like:

|2019-05-04T04:40:20.413119293Z- [-33400560.850078963, 11671074.30096948] 2019-05-04T04:40:20.413170584Zwarping 11 of 19 2019-05-04T04:40:20.573044610Zconvert -contrast-stretch 0 public/warps/193358-working/w193358-a_step_11_IMG_5313.JPG -crop 4287x4287+0+0! -flatten -distort Perspective '0,0 0,383 4287,0 99,0 4287,2821 314,122 0,2821 237,506' -flatten -crop 506x506+0+0! +repage public/warps/4621/w193358.png |

Wonder why?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-489342414 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AABQYS3ADNC27VDW2BPHVQ3PTW3HZANCNFSM4HKCHODA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-489355807, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J4RHLCOOIGMSGLCBELPTXMW3ANCNFSM4HKCHODA .

jywarren commented 5 years ago

I wonder if it's this line ... Is file.open the right way to upload the image?

https://github.com/publiclab/mapknitter-exporter-sinatra/blob/bdfb81602f5dd57e19d74953340c30714e05c489/app.rb#L172

jywarren commented 5 years ago

Maybe File.open().read instead? Hmm.

jywarren commented 5 years ago

Hmm, no, this says File.open is correct: http://fog.io/storage/

icarito commented 5 years ago

I think it needs a rescue block and to try again a few times

El sábado, 4 de mayo de 2019, Jeffrey Warren notifications@github.com escribió:

Hmm, no, this says File.open is correct: http://fog.io/storage/

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23#issuecomment-489364747, or mute the thread https://github.com/notifications/unsubscribe-auth/AABQYS3ARBO5NZCLSZ3TGGDPTXZW7ANCNFSM4HKCHODA .

-- Sebastian Silva http://somosazucar.org/

jywarren commented 5 years ago

https://github.com/publiclab/mapknitter-exporter-sinatra/pull/29 once merged is ready to be deployed too!

icarito commented 5 years ago

I think we are running into https://github.com/fog/fog-google/issues/416 I'll try their workaround!

jywarren commented 5 years ago

🎉

This big map fully exported (cropping bug tracking here: https://github.com/publiclab/mapknitter-exporter-sinatra/issues/32): https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json

~30 images of ~2mb each!

URL was this but it now says:

{"error":{"errors":[{"domain":"global","reason":"lockedDomainExpired","message":"Locked Domain Expired"}],"code":401,"message":"Locked Domain Expired"}}

so, i think these links aren't permanent, which is good to know.

jywarren commented 5 years ago

Lemon fair burn has crop issues too but seems OK otherwise, hard to tell:

image

image

And lincoln peak vineyard also completed, with cropping issues and some missing images:

image

image

However, it's GREAT to know that it can crank through 48 images of ~3mb each!!!

jywarren commented 5 years ago

Noting for optimization, from https://gis.stackexchange.com/a/44739:

While I don't know why GDAL provides this overlap in functionality, be sure to set the cache for gdalwarp to make it really fast:

# assuming 3G of cache here:
gdalwarp --config GDAL_CACHEMAX 3000 -wm 3000 $(list_of_tiffs) merged.tiff
icarito commented 5 years ago

Deployment is now automated to facilitate improving it!

El miércoles, 22 de mayo de 2019, Jeffrey Warren notifications@github.com escribió:

Noting for optimization, from https://gis.stackexchange.com/a/44739:

While I don't know why GDAL provides this overlap in functionality, be sure to set the cache for gdalwarp to make it really fast:

assuming 3G of cache here:

gdalwarp --config GDAL_CACHEMAX 3000 -wm 3000 $(list_of_tiffs) merged.tiff

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23?email_source=notifications&email_token=AABQYSYHMRJ5IWVJVMQAADDPWYO6RA5CNFSM4HKCHODKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWBBXHQ#issuecomment-495066014, or mute the thread https://github.com/notifications/unsubscribe-auth/AABQYSZL3EJA7OOMMMRTG2TPWYO6RANCNFSM4HKCHODA .

-- Sebastian Silva http://somosazucar.org/

jywarren commented 5 years ago

Awesome! So, I just push a new commit? Thanks!!!

On Thu, May 23, 2019, 9:59 AM Sebastian Silva notifications@github.com wrote:

Deployment is now automated to facilitate improving it!

El miércoles, 22 de mayo de 2019, Jeffrey Warren <notifications@github.com

escribió:

Noting for optimization, from https://gis.stackexchange.com/a/44739:

While I don't know why GDAL provides this overlap in functionality, be sure to set the cache for gdalwarp to make it really fast:

assuming 3G of cache here:

gdalwarp --config GDAL_CACHEMAX 3000 -wm 3000 $(list_of_tiffs) merged.tiff

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23?email_source=notifications&email_token=AABQYSYHMRJ5IWVJVMQAADDPWYO6RA5CNFSM4HKCHODKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWBBXHQ#issuecomment-495066014 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AABQYSZL3EJA7OOMMMRTG2TPWYO6RANCNFSM4HKCHODA

.

-- Sebastian Silva http://somosazucar.org/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23?email_source=notifications&email_token=AAAF6J6HMN2LJTM7K4ZS7JTPW2PILA5CNFSM4HKCHODKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWCJ3GI#issuecomment-495230361, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J2ZL3IMN3Q76OMQMQTPW2PILANCNFSM4HKCHODA .

jywarren commented 5 years ago

image

image

https://mapknitter.org/maps/otter-view-park-west -- @icarito this looks pretty darn good!

I see a couple edges which are not fuzzy, and so that means they were cut off partway through their composite processes. I'm wondering how it happened, but it could be either a crash partway through, OR it could be that their new bounding boxes are not properly calculated and they're just cropped wrong. I'll have to check the intermediate images to be sure. I'm closing the "missing images" and "cropping" issues and moving back here for more load testing.

jywarren commented 5 years ago

https://github.com/publiclab/mapknitter-exporter-sinatra/issues/39

jywarren commented 5 years ago

1558733443_1558733443

jywarren commented 5 years ago

Now testing https://mapknitter.org/maps/lincoln-peak-vineyard -- http://export.mapknitter.org/export?url=https://mapknitter.org/maps/lincoln-peak-vineyard/warpables.json&scale=30

http://export.mapknitter.org/id/1558800531/status.json

jywarren commented 5 years ago

It's at warping 19 of 48 now.

jywarren commented 5 years ago

Oh no, i started a new one. Maybe we should invent a way to cancel exports...

jywarren commented 5 years ago

And now, i try http://export.mapknitter.org/working/1558800531/w132014.tif (the 17th or 18th image) and get:

This site can’t be reached The webpage at http://export.mapknitter.org/working/1558800531/w132014.tif might be temporarily down or it may have moved permanently to a new web address.

However a few moments later it loads and i download the tiff.

jywarren commented 5 years ago

Looks like it might have been cut off: image

jywarren commented 5 years ago

This one was also slightly cut off: image

jywarren commented 5 years ago

It looks like at a stage earlier than the distortion and blending/cropping -- so in gdalwarp or gdal_translate, not in imagemagick convert. It's worth pulling logs for this.

jywarren commented 5 years ago

OK, so, having tracked this, a fix should resolve https://github.com/publiclab/mapknitter-exporter-sinatra/issues/39

jywarren commented 5 years ago

Just noting that this export: https://mapknitter-exports-warps.storage.googleapis.com/1559057443/status.json

Based on https://mapknitter.org/maps/usfws-restoration-bayou-bonfouca-southeast-survey (9 images)

Just output this at 9cm/px:

https://mapknitter-exports-warps.storage.googleapis.com/1559057443/1559057443.jpg

It's 8408x14912 pixels!!!! 🙌 in approx. 10 minutes.

jywarren commented 5 years ago

@tech4gt Take a look at this last export we just generated! 8408x14912 pixels!!!

jywarren commented 5 years ago

{"status_url":"https://mapknitter-exports-warps.storage.googleapis.com/1559057443/status.json","status":"complete","tms":"public/tms/1559057443/","geotiff":"public/warps/1559057443/1559057443.tif","zip":"public/tms/1559057443.zip","jpg":"public/warps/1559057443/1559057443.jpg","export_id":1559057443,"user_id":null,"size":"501.64MB","width":"8408","height":"14912","start_time":null,"run_time":null,"gem_version":"1.0.7","cm_per_pixel":9.0}

tech4GT commented 5 years ago

@jywarren Can you share the details on scaling with the applied scale also? I mean right now I am just generating a canvas by adding all the individual widths and heights, is there a better way to do it?

jywarren commented 5 years ago

Sure, i'll open an issue here: https://github.com/publiclab/image-sequencer-app/issues/14

On Tue, May 28, 2019 at 2:20 PM Varun Gupta notifications@github.com wrote:

@jywarren https://github.com/jywarren Can you share the details on scaling with the applied scale also? I mean right now I am just generating a canvas by adding all the individual widths and heights, is there a better way to do it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23?email_source=notifications&email_token=AAAF6JYS7P56C7RCKND6P7DPXVZTBA5CNFSM4HKCHODKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWM73II#issuecomment-496631201, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J7QKHRUZOQMX4AS5E3PXVZTBANCNFSM4HKCHODA .

jywarren commented 5 years ago

We'll soon be able to compare to https://github.com/publiclab/image-sequencer-app/issues/19 -- the pure JavaScript implementation!