Debugging stuck generating job?

[mestaritonttu]

I am trying to convert LibreOffice's Doxygen stuff. The job ran for 1-2 minutes and generated 2402 rst files, but then got stuck and I had to ctrl-c out of it. How could I find out what is making it go crazy?

If you want to have a look, I guess I could share the .xmls.

[vovkos]

Archived xmls would be nice, thanks. Which exact module of LibreOffice are we talking about?

[mestaritonttu]

Well it's all the modules :D Currently found through this index: https://docs.libreoffice.org/ Doxygen example: https://docs.libreoffice.org/accessibility/html/classes.html

Here are the xmls zipped: http://wikisend.com/download/186082/libo-xmls.zip

The last generated rst for me was: struct_com_sun_star_ucb_WelcomeDynamicResultSetStruct.rst

I used exclude_patterns += ['page_index.rst'] in my conf.py and then -f c_index.rst.in -D g_introFile=page_index.rst in the doxyrest command.

[vovkos]

The same number of generated .rst files, but different symptoms -- in my case I saw a Lua error (attempt to concatenate a nil value) followed by a crash. Investigating a crash under address sanitizer showed a heap memory corruption in lua_pushlstring. Memory corruption can manifest itself in multitude of ways, so I guess the lockdown you experienced on your machine is one of them.

Turned out, with this database we've hit a stack overflow in Lua -- after exhausting the default pre-allocated stack, Lua goes and writes beyond the end of buffer. There is a way to increase the stack size in Lua, but I need to take some time to think how to do it better (so we don't hit another stack overflow with an larger XML database).

In any case, I'm sure there will be a fix pretty soon. Stay tuned.

[mestaritonttu]

I did see such a nil value (I think) error before, but then I was not using -D g_introFile=page_index.rst and thought the case was somehow different.

Great to hear you could debug!

[vovkos]

The Lua stack overflow problem is solved. I decided to place stack checks before each lua_createtable -- each table creation means there will be more pushes right after that to populate table' elements, so it seems like the right place for a stack check.

Try with the new version (both axl and doxyrest repos must be updated; if you are using the bundle repo doxyrest_b, be sure to do git submodule update

Now, after Doxyrest and Sphinx complete their jobs and you open HTML you will see that a lot of items are unlinked (Sphinx will complain a lot about unresolved refs). That's because many links in the XML database contain uppercase letters and Sphinx requires lowercase-only. To combat that, add to Doxyfile:

CASE_SENSE_NAMES = NO

and then rebuild Doxygen XML database. I should probably update the README file regarding this issue...

Please close the issue after you confirm the stack overflow problem is gone.

[mestaritonttu]

Thank you very much. I confirm the issue is gone.

Yet, it only generates 2782 rst files. This is going into customer support territory and I feel I should be paying you.. Here are the newly generated xml files (with CASE_SENSE_NAMES = NO): http://wikisend.com/download/939294/libreoffice-xmls.zip

[vovkos]

It also generated 2782 .rst files on my machine. There is no one-to-one correspondence between the number of .xml and .rst files (Doxygen creates a dedicated .xml for each source file it processes, so there is always more .xml-s)

Have you run Sphinx after? It took 6 minutes to complete (here's Python for you), but the result seems alright; a few glitches here and there, but I'd say it's a good starting point.

[mestaritonttu]

Thanks for the clarification.

I tried to run Sphinx, but it fails with: Could not import extension doxyrest (exception: No module named 'doxyrest')

I have this at the end of my conf.py:

sys.path.insert(1, os.path.abspath('/home/tonttu/Documents/LibreOffice-jutut/doxyrest_b/build/doxyrest/bin/Release/')) extensions += ['doxyrest', 'cpplexer'] exclude_patterns += ['page_index.rst']

I tried to search for a solution, but could not find anything fitting.

Btw. at first I didn't realize that I have to uncomment import os and import sys as I am a total newbie to Sphinx. Might be good to add to the quick howto.

[vovkos]

Sphinx extensions are here: doxyrest/sphinx/*.py So just replace the first line with:

sys.path.insert(1, '/home/tonttu/Documents/LibreOffice-jutut/doxyrest_b/doxyrest/sphinx')

But you're right, it's worth adding to the HOWTO.

[mestaritonttu]

Oh boy, it worked! Sure, there were 3731 warnings, but it worked :)

Now to tweak the templates and then I can work with the LibreOffice developers and sysadmins to see, if we can start serving the htmls.

Thanks again for all the support.

[vovkos]

Sure, glad it helped! :)

[stansotn]

Sphinx extensions are here: doxyrest/sphinx/*.py So just replace the first line with:

sys.path.insert(1, '/home/tonttu/Documents/LibreOffice-jutut/doxyrest_b/doxyrest/sphinx')

But you're right, it's worth adding to the HOWTO.

Got stuck on the same silly issue, spent hours until I found this comment. +1 to adding it to HOWTO. Thanks for a handy tool, I much prefer it to the breathe tool=)

[vovkos]

Sorry for the late reply, I somehow missed your comment.

Got stuck on the same silly issue, spent hours until I found this comment. +1 to adding it to HOWTO.

Done.

Thanks for a handy tool, I much prefer it to the breathe tool=)

Thanks, I appreciate that! Glad it works for you!