search

mattvenn / cursivedata

polargraph energy monitor
1 stars 2 forks source link

pipeline fails with parsing svg #64

Open mattvenn opened 11 years ago

mattvenn commented 11 years ago

what happens is the pipeline and endpoint both end up with 0 length svg files. I thought it was due to an svg parsing issue, but looking at the logs not so sure. The 'no element found' happens a lot and is probably due to generators not creating any output.

grep 'no element' log/error.log | cut -d'[' -f3 | sort | uniq -c | sed -e 's/no element.*//' 33 error] >>cosm update failed! 1986 error] problem appending data/working/endpoint_1_partial.svg to data/working/endpoint_1_complete.svg: 689 error] problem appending data/working/endpoint_2_partial.svg to data/working/endpoint_2_complete.svg: 1764 error] problem appending data/working/endpoint_6_partial.svg to data/working/endpoint_6_complete.svg: 531 error] problem appending data/working/pipeline_2_partial.svg to data/working/pipeline_2_complete.svg:

In the case that just failed, the pipeline's id was 2 and so was the robot.

mattvenn commented 11 years ago

thought that this could be because parsing super big files can take a long time, and if the file is opened/being written while another update comes in - that could then be the cause. So have added file locking to the append_svg method of svg.py. Not handling the errors though, ideally will fail and the data will stay waiting in pipeline.

mattvenn commented 11 years ago

Reckon this has solved it. This from the logs:

mv:2161 [master]: tail -f log/error.log | grep 'lock in use' [Tue Jul 23 11:03:24 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock [Tue Jul 23 12:05:26 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock [Tue Jul 23 13:03:11 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock [Tue Jul 23 13:21:14 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock [Tue Jul 23 15:11:16 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock [Tue Jul 23 15:40:33 2013] [error] lock in use: /tmp/data.working.endpoint_2_complete.svg.lock

endpoitn2.complete.svg is currently a 700kb, 10000 line long svg file. It is often unavailable for viewing on the website, I suppose due to long times spent appending/parsing/writing.

Leaving the bug open because it still isn't handled well (exceptions), and it needs some kind of longer term solution for these files that can grow huge.

mattvenn commented 11 years ago

endpoint_2_complete.svg still got made a 0 length file somehow. But when I pressed reset on the endpoint, the stored output (that shows previous drawings) then became the complete svg....???

mattvenn commented 11 years ago

put some timings in, and for large files (20k lines, 1.5MB) it can take a minute or more to append and clean lines of the svg file. which is why the files seem missing. Also, it makes it almost inevitable that if the server is restarted the files are lost.

So, I suggest not bothering with endpoint svgs as we never use them. switch to pngs. Keep the pipeline svgs, but

1/- possibly only update them occasionally 2/- do the parsing/appending/cleaning on a tmp file and only mv it over the original if things go well.

ghost commented 11 years ago

another suggestion

1/- write a much faster appender that doesn't bother parsing xml 2/- use apache to gzip svg files to send them to client

mattvenn commented 11 years ago

replaced the pysvg parser with elementtree and got about a 600x speed increase. So now large files take seconds not minutes. Maybe that will fix things?