search

publiclab / image-sequencer-app

An Image Processing server based on image-sequencer
GNU General Public License v3.0
7 stars 5 forks source link

Load testing! #19

Open jywarren opened 5 years ago

jywarren commented 5 years ago

Now, building on the list from https://github.com/publiclab/mapknitter-exporter-sinatra/issues/23, let's see how big images/maps it can handle!

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json&scale=30 is real fast. seconds.

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/ceres--2/warpables.json&scale=30 -- It worked!!!! 25mb output in <2 minutes! Too big even to upload to GitHub! https://www.dropbox.com/s/m6v131cyynlwx0a/ceres--2.png?dl=0

(note that scale isn't working now -- #14 )

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 next:

Then:

jywarren commented 5 years ago

Hmm, i just saw an error on http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 , the URL became:

http://34.74.118.242/api/v2/process/?steps=[%7B%22id%22:1,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305302/G0097816.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2090%252C854%7Cne%3A2074%252C340%7Cse%3A2423%252C340%7Csw%3A2437%252C851%7D%22%7D,%7B%22id%22:2,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305303/G0131035.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3460%252C874%7Cne%3A3207%252C930%7Cse%3A3081%252C590%7Csw%3A3380%252C540%7D%22%7D,%7B%22id%22:3,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305305/G0131032.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3393%252C872%7Cne%3A3048%252C737%7Cse%3A3169%252C468%7Csw%3A3530%252C644%7D%22%7D,%7B%22id%22:4,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305306/G0131027.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3370%252C781%7Cne%3A3144%252C730%7Cse%3A3217%252C614%7Csw%3A3443%252C663%7D%22%7D,%7B%22id%22:5,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305307/G0131006.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3264%252C619%7Cne%3A3009%252C475%7Cse%3A3225%252C309%7Csw%3A3421%252C480%7D%22%7D,%7B%22id%22:6,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305308/G0097949.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2392%252C1149%7Cne%3A2015%252C1151%7Cse%3A2003%252C921%7Csw%3A2378%252C915%7D%22%7D,%7B%22id%22:7,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305310/G0097996.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2158%252C1213%7Cne%3A2034%252C752%7Cse%3A2466%252C788%7Csw%3A2491%252C1197%7D%22%7D,%7B%22id%22:8,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305311/G0131043.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3506%252C761%7Cne%3A3298%252C907%7Cse%3A2980%252C668%7Csw%3A3193%252C487%7D%22%7D,%7B%22id%22:9,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/305313/G0131096.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3015%252C859%7Cne%3A2922%252C594%7Cse%3A3203%252C556%7Csw%3A3277%252C821%7D%22%7D,%7B%22id%22:10,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306085/G0130831.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3133%252C640%7Cne%3A2934%252C490%7Cse%3A3099%252C356%7Csw%3A3303%252C476%7D%22%7D,%7B%22id%22:11,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306086/G0130807.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3436%252C373%7Cne%3A3448%252C609%7Cse%3A3080%252C678%7Csw%3A3068%252C441%7D%22%7D,%7B%22id%22:12,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306087/G0130809.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3491%252C477%7Cne%3A3393%252C731%7Cse%3A3005%252C576%7Csw%3A3141%252C338%7D%22%7D,%7B%22id%22:13,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306088/G0130799.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3542%252C751%7Cne%3A3197%252C860%7Cse%3A3057%252C582%7Csw%3A3399%252C472%7D%22%7D,%7B%22id%22:14,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306090/G0130944.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A3180%252C544%7Cne%3A2955%252C468%7Cse%3A2981%252C342%7Csw%3A3220%252C434%7D%22%7D,%7B%22id%22:15,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306091/G0130807.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2218%252C426%7Cne%3A2342%252C746%7Cse%3A1897%252C841%7Csw%3A1799%252C489%7D%22%7D,%7B%22id%22:16,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306092/G0120438.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2568%252C353%7Cne%3A1933%252C517%7Cse%3A1818%252C262%7Csw%3A2403%252C71%7D%22%7D,%7B%22id%22:17,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306093/G0120180.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1446%252C1456%7Cne%3A39%252C1582%7Cse%3A0%252C836%7Csw%3A1243%252C767%7D%22%7D,%7B%22id%22:18,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306097/G0098199.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2321%252C1505%7Cne%3A1868%252C1498%7Cse%3A1904%252C1249%7Csw%3A2349%252C1236%7D%22%7D,%7B%22id%22:19,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306098/G0098074_%282%29.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2441%252C1360%7Cne%3A2168%252C1344%7Cse%3A2079%252C1109%7Csw%3A2374%252C1056%7D%22%7D,%7B%22id%22:20,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306099/G0098249.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2236%252C1742%7Cne%3A1824%252C1418%7Cse%3A2220%252C1104%7Csw%3A2676%252C1528%7D%22%7D,%7B%22id%22:21,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306100/G0098286.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2188%252C1802%7Cne%3A1723%252C1616%7Cse%3A1946%252C1306%7Csw%3A2472%252C1639%7D%22%7D,%7B%22id%22:22,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306103/G0108574.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2492%252C2085%7Cne%3A2018%252C2087%7Cse%3A2036%252C1718%7Csw%3A2500%252C1727%7D%22%7D,%7B%22id%22:23,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306105/G0108666.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2055%252C2285%7Cne%3A2150%252C1898%7Cse%3A2634%252C1864%7Csw%3A2531%252C2315%7D%22%7D,%7B%22id%22:24,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306106/G0108717.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2173%252C2524%7Cne%3A1723%252C1876%7Cse%3A2350%252C1700%7Csw%3A2718%252C2355%7D%22%7D,%7B%22id%22:25,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306107/G01087970.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1961%252C2434%7Cne%3A1741%252C2158%7Cse%3A2011%252C2013%7Csw%3A2246%252C2302%7D%22%7D,%7B%22id%22:26,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306108/G0108827.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1922%252C2508%7Cne%3A1577%252C2345%7Cse%3A1713%252C2219%7Csw%3A2042%252C2376%7D%22%7D,%7B%22id%22:27,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306109/G0118976.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2031%252C2705%7Cne%3A1686%252C2545%7Cse%3A1800%252C2367%7Csw%3A2127%252C2524%7D%22%7D,%7B%22id%22:28,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306110/G0118987.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2075%252C2694%7Cne%3A1790%252C2729%7Cse%3A1754%252C2553%7Csw%3A2012%252C2529%7D%22%7D,%7B%22id%22:29,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306111/G0119015.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1848%252C2526%7Cne%3A1632%252C2646%7Cse%3A1503%252C2511%7Csw%3A1809%252C2410%7D%22%7D,%7B%22id%22:30,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306112/G0119224.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1730%252C2422%7Cne%3A1610%252C2225%7Cse%3A1765%252C2148%7Csw%3A1882%252C2345%7D%22%7D,%7B%22id%22:31,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306114/G0119266.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2090%252C2225%7Cne%3A1678%252C2112%7Cse%3A1756%252C1924%7Csw%3A2175%252C2044%7D%22%7D,%7B%22id%22:32,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306115/G0119306.jpg%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2232%252C1945%7Cne%3A1842%252C2110%7Cse%3A1681%252C1883%7Csw%3A2089%252C1703%7D%22%7D,%7B%22id%22:33,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306506/G0120190.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2227%252C1132%7Cne%3A1570%252C617%7Cse%3A2181%252C131%7Csw%3A2836%252C643%7D%22%7D,%7B%22id%22:34,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306507/G0120182.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2241%252C1337%7Cne%3A1509%252C633%7Cse%3A2232%252C115%7Csw%3A3033%252C884%7D%22%7D,%7B%22id%22:35,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306508/G0119327.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2122%252C1943%7Cne%3A1727%252C1788%7Cse%3A1906%252C1629%7Csw%3A2217%252C1729%7D%22%7D,%7B%22id%22:36,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306509/G0119418.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2521%252C1542%7Cne%3A1883%252C1679%7Cse%3A1808%252C1347%7Csw%3A2367%252C1186%7D%22%7D,%7B%22id%22:37,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306510/G0119445.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2528%252C1534%7Cne%3A1803%252C1552%7Cse%3A1784%252C1174%7Csw%3A2453%252C1106%7D%22%7D,%7B%22id%22:38,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306512/G0119577.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2325%252C1307%7Cne%3A1894%252C1330%7Cse%3A1883%252C1004%7Csw%3A2368%252C1056%7D%22%7D,%7B%22id%22:39,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306513/G0119782.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2499%252C139%7Cne%3A2031%252C570%7Cse%3A1642%252C331%7Csw%3A1890%252C0%7D%22%7D,%7B%22id%22:40,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306514/G0118977.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2367%252C2737%7Cne%3A2117%252C2589%7Cse%3A2289%252C2402%7Csw%3A2508%252C2572%7D%22%7D,%7B%22id%22:41,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306515/G0118957.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A2150%252C2637%7Cne%3A1910%252C2509%7Cse%3A2029%252C2359%7Csw%3A2270%252C2492%7D%22%7D,%7B%22id%22:42,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306516/G0119125.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1640%252C2506%7Cne%3A1445%252C2414%7Cse%3A1503%252C2258%7Csw%3A1768%252C2376%7D%22%7D,%7B%22id%22:43,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/306517/G0119127.JPG%22,%22depends%22:[],%22steps%22:%22webgl-distort%7Bnw%3A1634%252C2644%7Cne%3A1375%252C2389%7Cse%3A1591%252C2278%7Csw%3A1841%252C2489%7D%22%7D,%7B%22id%22:44,%22input%22:1,%22depends%22:[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],%22steps%22:%22canvas-resize%7Bwidth:7083.375%7Cheight:5474.75%7Cx:2074%7Cy:340%7D,import-image%7Burl:output%3E2%7D,overlay%7Bx:3081%7Cy:540%7D,import-image%7Burl:output%3E3%7D,overlay%7Bx:3048%7Cy:468%7D,import-image%7Burl:output%3E4%7D,overlay%7Bx:3144%7Cy:614%7D,import-image%7Burl:output%3E5%7D,overlay%7Bx:3009%7Cy:309%7D,import-image%7Burl:output%3E6%7D,overlay%7Bx:2003%7Cy:915%7D,import-image%7Burl:output%3E7%7D,overlay%7Bx:2034%7Cy:752%7D,import-image%7Burl:output%3E8%7D,overlay%7Bx:2980%7Cy:487%7D,import-image%7Burl:output%3E9%7D,overlay%7Bx:2922%7Cy:556%7D,import-image%7Burl:output%3E10%7D,overlay%7Bx:2934%7Cy:356%7D,import-image%7Burl:output%3E11%7D,overlay%7Bx:3068%7Cy:373%7D,import-image%7Burl:output%3E12%7D,overlay%7Bx:3005%7Cy:338%7D,import-image%7Burl:output%3E13%7D,overlay%7Bx:3057%7Cy:472%7D,import-image%7Burl:output%3E14%7D,overlay%7Bx:2955%7Cy:342%7D,import-image%7Burl:output%3E15%7D,overlay%7Bx:1799%7Cy:426%7D,import-image%7Burl:output%3E16%7D,overlay%7Bx:1818%7Cy:71%7D,import-image%7Burl:output%3E17%7D,overlay%7Bx:0%7Cy:767%7D,import-image%7Burl:output%3E18%7D,overlay%7Bx:1868%7Cy:1236%7D,import-image%7Burl:output%3E19%7D,overlay%7Bx:2079%7Cy:1056%7D,import-image%7Burl:output%3E20%7D,overlay%7Bx:1824%7Cy:1104%7D,import-image%7Burl:output%3E21%7D,overlay%7Bx:1723%7Cy:1306%7D,import-image%7Burl:output%3E22%7D,overlay%7Bx:2018%7Cy:1718%7D,import-image%7Burl:output%3E23%7D,overlay%7Bx:2055%7Cy:1864%7D,import-image%7Burl:output%3E24%7D,overlay%7Bx:1723%7Cy:1700%7D,import-image%7Burl:output%3E25%7D,overlay%7Bx:1741%7Cy:2013%7D,import-image%7Burl:output%3E26%7D,overlay%7Bx:1577%7Cy:2219%7D,import-image%7Burl:output%3E27%7D,overlay%7Bx:1686%7Cy:2367%7D,import-image%7Burl:output%3E28%7D,overlay%7Bx:1754%7Cy:2529%7D,import-image%7Burl:output%3E29%7D,overlay%7Bx:1503%7Cy:2410%7D,import-image%7Burl:output%3E30%7D,overlay%7Bx:1610%7Cy:2148%7D,import-image%7Burl:output%3E31%7D,overlay%7Bx:1678%7Cy:1924%7D,import-image%7Burl:output%3E32%7D,overlay%7Bx:1681%7Cy:1703%7D,import-image%7Burl:output%3E33%7D,overlay%7Bx:1570%7Cy:131%7D,import-image%7Burl:output%3E34%7D,overlay%7Bx:1509%7Cy:115%7D,import-image%7Burl:output%3E35%7D,overlay%7Bx:1727%7Cy:1629%7D,import-image%7Burl:output%3E36%7D,overlay%7Bx:1808%7Cy:1186%7D,import-image%7Burl:output%3E37%7D,overlay%7Bx:1784%7Cy:1106%7D,import-image%7Burl:output%3E38%7D,overlay%7Bx:1883%7Cy:1004%7D,import-image%7Burl:output%3E39%7D,overlay%7Bx:1642%7Cy:0%7D,import-image%7Burl:output%3E40%7D,overlay%7Bx:2117%7Cy:2402%7D,import-image%7Burl:output%3E41%7D,overlay%7Bx:1910%7Cy:2359%7D,import-image%7Burl:output%3E42%7D,overlay%7Bx:1445%7Cy:2258%7D,import-image%7Burl:output%3E43%7D,overlay%7Bx:1375%7Cy:2278%7D%22%7D]

This page isn’t working If the problem continues, contact the site owner. HTTP ERROR 400

jywarren commented 5 years ago

@tech4GT any ideas?

jywarren commented 5 years ago

Hmm. http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/lemon-fair-burn/warpables.json&scale=30 is also showing HTTP error 400... :-(

jywarren commented 5 years ago

Hmm, are we busting through memory here @icarito ? Where can we pull logs?

icarito commented 5 years ago

I use kubectl logs [container-name] to read logs. I get container-name from kubectl get pods

tech4GT commented 5 years ago

Okay, so the first thong about the url transforming is that right now everything is Implemented through GET and hence the redirect with the sequence as query. A rather small refactor will fix that, I'll push that as soon as I can. For the error, let's see the logs and then I might be able to figure out why it didn't work!

jywarren commented 5 years ago

@icarito I didn't see a new kubernetes container/cluster, which service did you set this up in? Thanks!!

On Mon, Jun 10, 2019, 8:16 PM Varun Gupta notifications@github.com wrote:

Okay, so the first thong about the url transforming is that right now everything is Implemented through GET and hence the redirect with the sequence as query. A rather small refactor will fix that, I'll push that as soon as I can. For the error, let's see the logs and then I might be able to figure out why it didn't work!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J2P7L7L4KCPBISZETTPZ3VDVA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXLSHNI#issuecomment-500638645, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J3BY3SVLHEJ6ZQQDZ3PZ3VDVANCNFSM4HWZEIJA .

icarito commented 5 years ago

imagen It's running in same cluster "mapknitter"

jywarren commented 5 years ago

Hmm, could be this error? 6/10/19, 5:58 PM

(node: 21) UnhandledPromiseRejectionWarning: Error: Protocol error (Runtime.callFunctionOn): Target closed.
at Promise (/app/node_modules/puppeteer/lib/Connection.js:183)
at Promise (<anonymous>)
at CDPSession.send (/app/node_modules/puppeteer/lib/Connection.js:182)
at ExecutionContext.evaluateHandle (/app/node_modules/puppeteer/lib/ExecutionContext.js:106)
at ExecutionContext.<anonymous> (/app/node_modules/puppeteer/lib/helper.js:111)
at ExecutionContext.evaluate (/app/node_modules/puppeteer/lib/ExecutionContext.js:48)
at ExecutionContext.<anonymous> (/app/node_modules/puppeteer/lib/helper.js:111)
at DOMWorld.evaluate (/app/node_modules/puppeteer/lib/DOMWorld.js:112)
at process._tickCallback (next_tick.js:68)
jywarren commented 5 years ago

Yeah another log clue

textPayload:  "    at page.addScriptTag.then (/app/node_modules/image-sequencer/src/modules/_nomodule/gl-context.js:18:16)
"
jywarren commented 5 years ago 
(node:21) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
jywarren commented 5 years ago

https://console.cloud.google.com/logs/viewer?resource=k8s_container%2Fcluster_name%2Fmapknitter%2Fnamespace_name%2Fdefault%2Fcontainer_name%2Fimage-sequencer-app&logName=projects%2Fpublic-lab%2Flogs%2Fstderr&expandAll=false&timestamp=2019-06-10T21:58:41.154000000Z&project=public-lab&authuser=1&minLogLevel=0&customFacets=&limitCustomFacetWidth=true&interval=JUMP_TO_TIME&scrollTimestamp=2019-06-10T21:57:32.444075029Z

tech4GT commented 5 years ago

@jywarren Could you try running this with v1 API, it'll take considerably more time, but takes far less memory. That should confirm if the problem is being caused by memory constraints.

jywarren commented 5 years ago

I also see, near and after that last error, "Finished!Finished!Finished!..."

jywarren commented 5 years ago

I will tomorrow! Need to sleep now :-)

jywarren commented 5 years ago

Error catching can be done here too! https://console.cloud.google.com/errors/CM_GmvOcrOX8_QE?time=P1D&project=public-lab&authuser=1

tech4GT commented 5 years ago

Oh! Good night! Also I think we might need to do something about the logs on image sequencer. Maybe I'll refactor that over the weekend.

jywarren commented 5 years ago

Hmm, @tech4GT when i go to http://34.74.118.242/api/v1/export/?url=http://mapknitter.org/maps/lemon-fair-burn/warpables.json&scale=30, it redirects to:

http://34.74.118.242/api/v1/process/?steps=[%7B%22id%22:1,%22input%22:%22https://s3.amazonaws.com/grassrootsmapping/warpables/87835/Lemon-20120420-125-91.jpg%22,%22steps%22:%22webgl-distort%7Bnw%3A2201%252C2137%7Cne%3A2553%252C2245%7Cse%3A2531%252C2851%7Csw%3A2157%252C2696%7D,canvas-resize%7Bwidth:4562.5%7Cheight:4420%7Cx:2157%7Cy:2137%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87839/Lemon-20120420-125-92.jpg%7D,webgl-distort%7Bnw%3A2435%252C2711%7Cne%3A2186%252C2867%7Cse%3A2091%252C2152%7Csw%3A2340%252C1997%7D,overlay%7Bx:2091%7Cy:1997%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87843/Lemon-20120420-125-16.jpg%7D,webgl-distort%7Bnw%3A1336%252C693%7Cne%3A1218%252C1702%7Cse%3A881%252C1360%7Csw%3A1013%252C522%7D,overlay%7Bx:881%7Cy:522%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87847/Lemon-20120420-125-19.jpg%7D,webgl-distort%7Bnw%3A422%252C886%7Cne%3A0%252C683%7Cse%3A50%252C0%7Csw%3A503%252C96%7D,overlay%7Bx:0%7Cy:0%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87851/Lemon-20120420-125-17.jpg%7D,webgl-distort%7Bnw%3A639%252C1270%7Cne%3A448%252C1764%7Cse%3A273%252C1461%7Csw%3A464%252C966%7D,overlay%7Bx:273%7Cy:966%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87855/Lemon-20120420-125-21.jpg%7D,webgl-distort%7Bnw%3A1094%252C1391%7Cne%3A764%252C1283%7Cse%3A859%252C305%7Csw%3A1196%252C460%7D,overlay%7Bx:764%7Cy:305%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87859/Lemon-20120420-125-23.jpg%7D,webgl-distort%7Bnw%3A595%252C755%7Cne%3A793%252C87%7Cse%3A1196%252C600%7Csw%3A1042%252C1221%7D,overlay%7Bx:595%7Cy:87%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87863/Lemon-20120420-125-24.jpg%7D,webgl-distort%7Bnw%3A1061%252C2077%7Cne%3A868%252C2350%7Cse%3A696%252C1805%7Csw%3A889%252C1532%7D,overlay%7Bx:696%7Cy:1532%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87867/Lemon-20120420-125-27.jpg%7D,webgl-distort%7Bnw%3A607%252C2396%7Cne%3A410%252C2204%7Cse%3A531%252C1649%7Csw%3A727%252C1841%7D,overlay%7Bx:410%7Cy:1649%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87871/Lemon-20120420-125-29.jpg%7D,webgl-distort%7Bnw%3A1666%252C1562%7Cne%3A1123%252C1795%7Cse%3A1138%252C1019%7Csw%3A1541%252C739%7D,overlay%7Bx:1123%7Cy:739%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87875/Lemon-20120420-125-32.jpg%7D,webgl-distort%7Bnw%3A1235%252C2888%7Cne%3A972%252C2581%7Cse%3A1266%252C1850%7Csw%3A1460%252C2164%7D,overlay%7Bx:972%7Cy:1850%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87879/Lemon-20120420-125-33.jpg%7D,webgl-distort%7Bnw%3A1306%252C646%7Cne%3A1754%252C926%7Cse%3A1592%252C1593%7Csw%3A1145%252C1391%7D,overlay%7Bx:1145%7Cy:646%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87883/Lemon-20120420-125-34.jpg%7D,webgl-distort%7Bnw%3A1717%252C1376%7Cne%3A1446%252C1919%7Cse%3A1167%252C1236%7Csw%3A1453%252C770%7D,overlay%7Bx:1167%7Cy:770%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87887/Lemon-20120420-125-60.jpg%7D,webgl-distort%7Bnw%3A1937%252C1547%7Cne%3A1446%252C2028%7Cse%3A1372%252C1360%7Csw%3A1878%252C988%7D,overlay%7Bx:1372%7Cy:988%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87891/Lemon-20120420-125-62.jpg%7D,webgl-distort%7Bnw%3A1981%252C2478%7Cne%3A1424%252C1780%7Cse%3A1724%252C1205%7Csw%3A2142%252C1562%7D,overlay%7Bx:1424%7Cy:1205%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87895/Lemon-20120420-125-64.jpg%7D,webgl-distort%7Bnw%3A1988%252C2230%7Cne%3A1717%252C1516%7Cse%3A2135%252C988%7Csw%3A2252%252C1578%7D,overlay%7Bx:1717%7Cy:988%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87899/Lemon-20120420-125-65.jpg%7D,webgl-distort%7Bnw%3A2377%252C1919%7Cne%3A2157%252C2245%7Cse%3A2047%252C1469%7Csw%3A2296%252C1267%7D,overlay%7Bx:2047%7Cy:1267%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87903/Lemon-20120420-125-63.jpg%7D,webgl-distort%7Bnw%3A2069%252C2292%7Cne%3A1504%252C1749%7Cse%3A1842%252C1159%7Csw%3A2157%252C1562%7D,overlay%7Bx:1504%7Cy:1159%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87907/Lemon-20120420-125-66.jpg%7D,webgl-distort%7Bnw%3A2707%252C1966%7Cne%3A2384%252C2587%7Cse%3A2215%252C2013%7Csw%3A2509%252C1438%7D,overlay%7Bx:2215%7Cy:1438%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87911/Lemon-20120420-125-67.jpg%7D,webgl-distort%7Bnw%3A2677%252C2634%7Cne%3A2384%252C1997%7Cse%3A2597%252C1500%7Csw%3A2839%252C2090%7D,overlay%7Bx:2384%7Cy:1500%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87915/Lemon-20120420-125-69.jpg%7D,webgl-distort%7Bnw%3A2765%252C1702%7Cne%3A2927%252C2075%7Cse%3A2802%252C2447%7Csw%3A2626%252C2075%7D,overlay%7Bx:2626%7Cy:1702%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87919/Lemon-20120420-125-70.jpg%7D,webgl-distort%7Bnw%3A3154%252C1904%7Cne%3A3073%252C2680%7Cse%3A2802%252C2478%7Csw%3A2890%252C1764%7D,overlay%7Bx:2802%7Cy:1764%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87927/Lemon-20120420-125-73.jpg%7D,webgl-distort%7Bnw%3A3012%252C3133%7Cne%3A2784%252C2888%7Cse%3A2938%252C2242%7Csw%3A3167%252C2487%7D,overlay%7Bx:2784%7Cy:2242%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/87931/Lemon-20120420-125-74.jpg%7D,webgl-distort%7Bnw%3A3056%252C2092%7Cne%3A3424%252C2504%7Cse%3A3278%252C3090%7Csw%3A2909%252C2677%7D,overlay%7Bx:2909%7Cy:2092%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88599/Lemon-20120420-125-83.jpg%7D,webgl-distort%7Bnw%3A2715%252C2922%7Cne%3A2739%252C2392%7Cse%3A3073%252C2461%7Csw%3A3049%252C2990%7D,overlay%7Bx:2715%7Cy:2392%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88619/Lemon-20120420-125-86.jpg%7D,webgl-distort%7Bnw%3A2747%252C3182%7Cne%3A2521%252C2947%7Cse%3A2669%252C2309%7Csw%3A2895%252C2544%7D,overlay%7Bx:2521%7Cy:2309%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88623/Lemon-20120420-125-87.jpg%7D,webgl-distort%7Bnw%3A2839%252C2562%7Cne%3A2665%252C3143%7Cse%3A2460%252C2867%7Csw%3A2634%252C2286%7D,overlay%7Bx:2460%7Cy:2286%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88695/Lemon-20120420-125-93.jpg%7D,webgl-distort%7Bnw%3A2388%252C2723%7Cne%3A2128%252C2759%7Cse%3A2116%252C2012%7Csw%3A2398%252C1981%7D,overlay%7Bx:2116%7Cy:1981%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88699/Lemon-20120420-125-64.jpg%7D,webgl-distort%7Bnw%3A1487%252C3420%7Cne%3A1279%252C2797%7Cse%3A1725%252C2265%7Csw%3A1861%252C2828%7D,overlay%7Bx:1279%7Cy:2265%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88703/Lemon-20120420-125-35.jpg%7D,webgl-distort%7Bnw%3A2126%252C2615%7Cne%3A1655%252C1917%7Cse%3A1812%252C1078%7Csw%3A2178%252C1165%7D,overlay%7Bx:1655%7Cy:1078%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88711/Lemon-20120420-125-42.jpg%7D,webgl-distort%7Bnw%3A1827%252C1183%7Cne%3A1198%252C935%7Cse%3A1608%252C14%7Csw%3A1977%252C696%7D,overlay%7Bx:1198%7Cy:14%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88715/Lemon-20120420-125-43.jpg%7D,webgl-distort%7Bnw%3A1787%252C799%7Cne%3A2177%252C555%7Cse%3A2168%252C1408%7Csw%3A1744%252C1356%7D,overlay%7Bx:1744%7Cy:555%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88719/Lemon-20120420-125-44.jpg%7D,webgl-distort%7Bnw%3A1922%252C339%7Cne%3A2179%252C929%7Cse%3A1857%252C1432%7Csw%3A1714%252C844%7D,overlay%7Bx:1714%7Cy:339%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88723/Lemon-20120420-125-77.jpg%7D,webgl-distort%7Bnw%3A3563%252C2254%7Cne%3A3337%252C2931%7Cse%3A3083%252C2541%7Csw%3A3305%252C1922%7D,overlay%7Bx:3083%7Cy:1922%7Coffset:-3%7D,import-image%7Burl:https://s3.amazonaws.com/grassrootsmapping/warpables/88727/Lemon-20120420-125-79.jpg%7D,webgl-distort%7Bnw%3A3476%252C2790%7Cne%3A3050%252C3166%7Cse%3A3010%252C2459%7Csw%3A3375%252C2288%7D,overlay%7Bx:3010%7Cy:2288%7Coffset:-3%7D%22%7D]

Redirecting is not a problem - we can leave that -- but this second page doesn't work; i get a 400 error.

tech4GT commented 5 years ago

I'll look into this! Let me try it with a smaller test first!

tech4GT commented 5 years ago

Okay so the v2 api seems to be working with http://34.74.118.242/api/v2/export/?url=https://mapknitter.org/maps/ceres--2/warpables.json This does make me feel like the issue with larger exports might be memory!

jywarren commented 5 years ago

Hmm, yes indeed - this looks like it running, but I am still getting a 400... odd!

[Trimmed]0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s0h 0m 21s

Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing: Processing:    18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%18.33%  3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170 3,979,170/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992/21,708,992 Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: Elapsed: 0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s0h 0m 4s Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: Remaining: 0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m 20s0h 0m
tech4GT commented 5 years ago

Okay @jywarren I tried the v1 and that seems to work on the ceres map too!! http://34.74.118.242/api/v1/export/?url=https://mapknitter.org/maps/ceres--2/warpables.json

tech4GT commented 5 years ago

Okay we really need to take care of the logs at image sequencer! I'll take care of it this weekend!! So @jywarren I think the issue here might just be with the amount of memory! Since I have not placed a limit to the number of steps that can run concurrently, so it might be that the system just initiates a bunch of steps at the same time but ends up running out of memory to process them later. I think what we should do is put some sort of upper limit as to how many steps can run concurrently. What do you think?

jywarren commented 5 years ago

Like, lots of these are running. So, I wonder if we are in fact running all the steps, but somehow the original request is receiving a 400?

tech4GT commented 5 years ago

We can base the limit on the amount of available memory!

tech4GT commented 5 years ago

Like, lots of these are running. So, I wonder if we are in fact running all the steps, but somehow the original request is receiving a 400?

Hmm, this is with the v1, correct?

jywarren commented 5 years ago

Great to hear on the ceres map! OK, yeah. We could allow 3 or 5 processes to run concurrently, because in some of these maps we'd be starting 45 processes at once. So this will serialize them a bit?

jywarren commented 5 years ago

I'm not sure which actually, i just searched all recent logs on the cluster. It's odd, i can't see the original GET requests themselves. Maybe we need to output to stdout (from the is-app) some request information (like the request path, full parameters, etc) so that we can read the logs a bit more easily?

jywarren commented 5 years ago

We can base the limit on the amount of available memory!

great!

tech4GT commented 5 years ago

Great to hear on the ceres map! OK, yeah. We could allow 3 or 5 processes to run concurrently, because in some of these maps we'd be starting 45 processes at once. So this will serialize them a bit?

Let me do some testing and figure out the mapping b/w the memory and the number of steps running. Let's not hard code it though, what say?

tech4GT commented 5 years ago

I'm not sure which actually, i just searched all recent logs on the cluster. It's odd, i can't see the original GET requests themselves. Maybe we need to output to stdout (from the is-app) some request information (like the request path, full parameters, etc) so that we can read the logs a bit more easily?

I agree! Logs on Image sequencer need major work! The progress bars are also really obtrusive, fixing that alone should give us some performance boost!

tech4GT commented 5 years ago

Also @jywarren Just to be clear, right now the different steps are not actually run as separate processes, but are run concurrently. So to be very specific in v2 api we have concurrency not parallelism.

jywarren commented 5 years ago

And, if the request takes like 1 hour to process, shouldn't we follow this sequence -- as we do in the sinatra app:

  1. initiate export on starting GET or POST request
  2. fork off the export process and assign it an ID (like a pid or timestamp?)
  3. report back a URL with a status ID to the original request very quickly (like immediately)
  4. client can then hit a 2nd url like /api/v2/status/?pid=000000000 as often as it likes to learn what the status is

In https://github.com/publiclab/mapknitter-exporter-sinatra/, the status is actually written to a Google Cloud Storage status.json file, so there is no overhead to hitting it as many times as we like. But it doesn't have to be that way; we did it that way because when you send a 2nd request, you might not get the same instance responding, so if you store the process locally, it's only accessible to the container instance that created it.

jywarren commented 5 years ago

OK thanks for clarification on concurrency!

Let me do some testing and figure out the mapping b/w the memory and the number of steps running. Let's not hard code it though, what say?

This sounds great; you could even make it a GET parameter.

jywarren commented 5 years ago

Let's do the memory stuff first, and think about a 2nd request for status.json next, though!

tech4GT commented 5 years ago

Got it! My weekend checklist: 1> Fix logs on image sequencer 2> Update the process to confine the memory used 3> Fork the exporter process and add a status endpoint

Anything else you would like to addd @jywarren ??

jywarren commented 5 years ago

That sound amazing!

On Tue, Jun 11, 2019 at 10:58 AM Varun Gupta notifications@github.com wrote:

Got it! My weekend checklist: 1> Fix logs on image sequencer 2> Update the process to confine the memory used 3> Fork the exporter process and add a status endpoint

Anything else you would like to addd @jywarren https://github.com/jywarren ??

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J7Z767Y2SZB5P2GYZ3PZ64O5A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNNHXA#issuecomment-500880348, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J3BQMRVOJYX4OJ2O2LPZ64O5ANCNFSM4HWZEIJA .

jywarren commented 5 years ago

Linking back to https://github.com/publiclab/image-sequencer-app/issues/19#issuecomment-500693625 where @Divy123 has what appears to be the exact same error on:

Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit

tech4GT commented 5 years ago

Okay so the eventEmitter memory leak error, is because of setting more than 10 listeners on an event. I am not sure what code originates this but let's try if increasing the limit fixes that. I am pushing in some preliminary fixes which bounds the concurrent processes to 10. Since I can't test this on my local machine, let's try it on the server and monitor the logs for a huge export!

tech4GT commented 5 years ago

Also @jywarren Do we some place we can store these exports, since we are forking the process, eventually when the export completes we would have to send it to the user somehow. How is this done on the rails version?

tech4GT commented 5 years ago

@icarito @jywarren Please deploy the the new code I have pushed in and let's see what impact is it causing!

jywarren commented 5 years ago

In rails it is sent along with a status.json file to Google cloud storage. We can set that up too.

Is it building automatically now from the main branch push?

On Sat, Jun 15, 2019, 8:48 AM Varun Gupta notifications@github.com wrote:

@icarito https://github.com/icarito @jywarren https://github.com/jywarren Please deploy the the new code I have pushed in and let's see what impact is it causing!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J74MYXFKY35HMQN5FDP2TQHFA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYXPSY#issuecomment-502364107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J3EGUCPYWH6QMPPGTLP2TQHFANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

I’m not sure maybe @icarito would know?

jywarren commented 5 years ago

It built 20m ago automatically from the push!

On Sat, Jun 15, 2019, 9:06 AM Jeffrey Warren jeff@unterbahn.com wrote:

In rails it is sent along with a status.json file to Google cloud storage. We can set that up too.

Is it building automatically now from the main branch push?

On Sat, Jun 15, 2019, 8:48 AM Varun Gupta notifications@github.com wrote:

@icarito https://github.com/icarito @jywarren https://github.com/jywarren Please deploy the the new code I have pushed in and let's see what impact is it causing!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J74MYXFKY35HMQN5FDP2TQHFA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYXPSY#issuecomment-502364107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J3EGUCPYWH6QMPPGTLP2TQHFANCNFSM4HWZEIJA .

tech4GT commented 5 years ago

Awesome! That makes it so much better! I’ll test it out! Also if you’ll send me the credentials to upload to the cloud storage, I can set that up too!

jywarren commented 5 years ago

Hmm this still gives 400:

http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30

I'll check logs but maybe we need to try a big map with fewer than 10 images???

tech4GT commented 5 years ago

Can you point me to where I can see the logs??

jywarren commented 5 years ago

It's in Google storage logs, here:

https://console.cloud.google.com/logs/viewer?project=public-lab&resource=k8s_container%2Fcluster_name%2Fmapknitter&pli=1&authuser=1&minLogLevel=0&expandAll=false&timestamp=2019-06-15T13:21:28.976000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeStart=2019-06-15T12:21:29.228Z&dateRangeEnd=2019-06-15T13:21:29.228Z&interval=PT1H&scrollTimestamp=2019-06-15T13:20:48.634749144Z

But you may need to change the filters!

On Sat, Jun 15, 2019, 9:15 AM Varun Gupta notifications@github.com wrote:

Can you point me to where I can see the logs??

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J4XSRMI2FAI4YUJC2DP2TTNXA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYX6KA#issuecomment-502365992, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J4LHKPNZERO3VGYHITP2TTNXANCNFSM4HWZEIJA .

jywarren commented 5 years ago

Also I made a new bucket "mapknitter-is" that you have access to. You'll have to make yourself a key!

On Sat, Jun 15, 2019, 9:21 AM Jeffrey Warren jeff@unterbahn.com wrote:

It's in Google storage logs, here:

https://console.cloud.google.com/logs/viewer?project=public-lab&resource=k8s_container%2Fcluster_name%2Fmapknitter&pli=1&authuser=1&minLogLevel=0&expandAll=false&timestamp=2019-06-15T13:21:28.976000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeStart=2019-06-15T12:21:29.228Z&dateRangeEnd=2019-06-15T13:21:29.228Z&interval=PT1H&scrollTimestamp=2019-06-15T13:20:48.634749144Z

But you may need to change the filters!

On Sat, Jun 15, 2019, 9:15 AM Varun Gupta notifications@github.com wrote:

Can you point me to where I can see the logs??

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J4XSRMI2FAI4YUJC2DP2TTNXA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYX6KA#issuecomment-502365992, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J4LHKPNZERO3VGYHITP2TTNXANCNFSM4HWZEIJA .

jywarren commented 5 years ago

https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload

https://cloud.google.com/storage/docs/uploading-objects

On Sat, Jun 15, 2019, 9:22 AM Jeffrey Warren jeff@unterbahn.com wrote:

Also I made a new bucket "mapknitter-is" that you have access to. You'll have to make yourself a key!

On Sat, Jun 15, 2019, 9:21 AM Jeffrey Warren jeff@unterbahn.com wrote:

It's in Google storage logs, here:

https://console.cloud.google.com/logs/viewer?project=public-lab&resource=k8s_container%2Fcluster_name%2Fmapknitter&pli=1&authuser=1&minLogLevel=0&expandAll=false&timestamp=2019-06-15T13:21:28.976000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeStart=2019-06-15T12:21:29.228Z&dateRangeEnd=2019-06-15T13:21:29.228Z&interval=PT1H&scrollTimestamp=2019-06-15T13:20:48.634749144Z

But you may need to change the filters!

On Sat, Jun 15, 2019, 9:15 AM Varun Gupta notifications@github.com wrote:

Can you point me to where I can see the logs??

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer-app/issues/19?email_source=notifications&email_token=AAAF6J4XSRMI2FAI4YUJC2DP2TTNXA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYX6KA#issuecomment-502365992, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J4LHKPNZERO3VGYHITP2TTNXANCNFSM4HWZEIJA .