search

HaxeFoundation / haxelib

The Haxe library manager
https://lib.haxe.org/
MIT License
174 stars 77 forks source link

haxelib submit freezes on "Processing file...." #201

Open jgranick opened 9 years ago

jgranick commented 9 years ago

Recently, I have not been able to submit haxelibs (or if I am, I'm lucky for it to work before a very long time out)

This appears to be related to including a "haxedoc.xml", as this occurs for OpenFL and Lime, but not for smaller libraries that don't include a generated doc XML file

back2dos commented 9 years ago

Could you provide a sample archive to reproduce the problem?

jgranick commented 9 years ago

Try submitting a recent version of Lime or OpenFL, or if I understand the issue, an archive with one of their "haxedoc.xml" files

ibilon commented 8 years ago

Having a haxedoc.xml will trigger this https://github.com/HaxeFoundation/haxelib/blob/d3593aeb315cef8be6aef1a53e4fc5e932181e1f/src/haxelib/server/Repo.hx#L235-L262

but I don't think there's anything heavy going on, unless the xml is heavy? @jgranick

Also the Data function returns the first haxedoc.xml, regardless of how deep it is, maybe we would only look at it if it's next to haxelib.json?

ibilon commented 8 years ago

Alright so I tested it with the zip from openfl 3.0.1, had to remove all but haxelib.json and a third of haxedoc.xml (nekotools server as a very low post size limit) and the server took 6 minutes at 100% cpu.

I looked deeper at the code I linked, and it actually sounds quite heavy if you have a lot of types. Processing, sorting, copying and serializing a 2Mb xml file isn't quick.

Do we really need to parse the documentation? I can't even find where to access it from the website.

nadako commented 8 years ago

If anything, this should be done in a separate process, not in critical upload handler.

ibilon commented 8 years ago

I checked for .documentation in the entire repository: in the current website: nothing in the current server/client: only two mention in server when processing an upload the legacy server had a page to show the documentation.

So either we should show it, wonder why it didn't make it into the new website (and offload the parsing after the submit) or stop parsing it.