dlang / ddox

Advanced D documentation engine
MIT License
63 stars 21 forks source link

Unicode quotes in comments causes ddox to hang #109

Closed gnunn1 closed 8 years ago

gnunn1 commented 8 years ago

I'm fooling around with getting ddox working with the GtkD source code base, one thing I've noticed is that ddox hangs on certain GtkD classes. Investigating further, the problem appears to be related to the usage of Unicode quotes in GtkD comments, i.e. the “ and ” quotes.

For example, the comment in gdk.Window.invalidateMaybeRecurse method (https://github.com/gtkd-developers/GtkD/blob/master/src/gdk/Window.d) uses these quotes around the term "dirty region". Replacing the unicode quotes with ASCII quotes fixes the problem and ddox no longer hangs.

gnunn1 commented 8 years ago

Looks like Unicode characters in general cause problems, I was able to work around it by converting the docs.json file to ASCII using the following script. The script converts single and quote characters to their ASCII equivalent using sed before converting everything else using iconv.

#Fix problem with unicode quotes by replacing them with ASCII quotes
echo Replacing unicode double and single quotes with ASCII equivalent
sed -i 's/“/\"/g' ddox/docs.json
sed -i 's/”/\"/g' ddox/docs.json
sed -i 's/’/\'/g' ddox/docs.json

echo "Convert UTF-8 to ASCII for everything else"
mv ddox/docs.json ddox/docs_utf8.json
iconv -f utf8 -t ascii -c ddox/docs_utf8.json > ddox/docs.json