Exporting pages to Single Page Html with Indexmenu created TOC page

[rgleason]

Set Opencpn Manual on the page https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:toc Then goto "Admin" > Additional Plugins > SiteExportManager

Set Namespace: opencpn:opencpn_user_manual:toc Parent Namespace to export:opencpn:opencpn_user_manual:toc Export Linked Pages: specified depth Depth:3 need down to 7 Export Linked Pages: Checked Render Engine: ckgedit Export Template: bootstrap3 Numbered Headings: checked Use TOC file in Namespace: unchecked Empty Namespaces in TOC: checked Curl L used for download

SiteExport works to a depth of 3. When I set it on Depth of 4 we get timeout messages. Sometimes they are on page 20, 30 or 72 of a total of 143 pages. It has never completed. Should we look for errors in the pages or lengthen the timeout period. (I believe we lengthened it last time too.) Found this in php.ini _SERVER["REQUEST_TIME_FLOAT"] | 1515120311.1936 _SERVER["REQUEST_TIME"] | 1515120311

I don't know if the pages have some problem, but we can create large siteexport.zip of 72 mb (160mb expanded) of the 360 pages using dw2pdf that download fine. We prefer the single page html format that uses a browser, and it is easier to edit.

Attached is the result - siteexport.zip. which only goes one level down apparently. The lower level pages are missing.

Do you have any suggestions for us? siteexport(8).zip

The User Manual pages we need for the next software v4.8.2 release are here: https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual

The IndexMenu created TOC , titled "Opencpn Embedded User Manual" here: https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:toc

[rgleason]

PS: I can export 143 pages down to level 4 using dw2pdf to make a siteexport.zip 37mb with bunch of PDF files in directories, but the links all go back to the original site.

Also I've been trying to get debug3.txt working, see https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:debug3.txt I have that listed in config for siteexport, and does allowdebug have to be checked too? plugin»siteexport»debugLevel debug plugin»siteexport»debugFile debug3.txt plugin»siteexport»cachetime 606024 Haven't seen any errors yet. Even with fails. Don't think I have it working yet.

[gamma]

Hi Rick, you said that you are using the curl command to create the siteexport.zip. Have you tried the Export button just below the lines? The exporter then will fetch page by page, each with a distinct timeout.

The debug file will not be a page. You can define a full system path there. I usually use /tmp/siteexport.txt - it might help send me this file - I'd suggest via email to tools@inetsoftware.de

[rgleason]

Gerry, What I have been doing is selecting "direct" "wget" or "curl" button, then selecting "Start" This time I selected "curl" then "Start" after setting Depth at "7" and going into Admin > Config to set the debug path & file to /tmp/siteexport.txt .

It timed out again.

Adding "opencpn:opencpn_user_manual:charts:vector_display_tab" ( 47 / 361 )
500 Internal Server Error Request Timeout
This request takes too long to process,  it is timed out by the server. 
If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

Just hitting "Start" button without selecting "Direct" "Wget" or "Curl" first.... (I think these have max redirects =4) When I set Depth to "1" or "2" it fails. When I set Depth to "3" it completes. When I set Depth to "4" it always fails.

Adding "opencpn:opencpn_user_manual:advanced_features:portable_opencpn_v2" ( 16 / 143 )
500 Internal Server Error Request Timeout
This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

Need to collect the Debug file, I hope.
Dave has given me today to get this done, before he has to revert the website back to PHP 5.3 because parts of the website only work with that.

[gamma]

Sorry to hear that. I think that the page might be too large has to expensive content rendering wise ... but that is just what I think without knowing it.

Because of the revert: you could try downloading an older version of the plugin. There are various releases that should support 5.3. In fact I just recently updated the plugin for newer versions of PHP.

[rgleason]

Well, the day is gone and I haven't got it working. I guess I'd best go back to an earlier version that works with PHP 5.3 or 5.4. How would I do that and which version?

Something like this one? https://github.com/i-net-software/dokuwiki-plugin-siteexport/commit/55c869b6a74bad0804df55a3b9eeaa77077dbe4d

I should probably know this by now, but where do I find the travis compile, so I can install manually? Thanks

[gamma]

check out the Release page. There are several releases there - I'll need to take some time to put a specific version to dokuwiki.org that will work with previous php versions.

You can copy/paste the releases zip url into your dokuwiki plugin manager to install a specific version.

https://github.com/i-net-software/dokuwiki-plugin-siteexport/releases

[rgleason]

Thanks Gerry, this one from March 23, 2017 still has 5.3 in travis

https://github.com/i-net-software/dokuwiki-plugin-siteexport/commit/93f9765070ac074b28ac778b97d1ca80644746f9

https://github.com/i-net-software/dokuwiki-plugin-siteexport/blob/8c592f422cc3619ec918f0839db6f7a02e13e9e8/.travis.yml

language: php php:

• "5.6"
• "5.5"
• "5.4"
• "5.3" env:
• DOKUWIKI=master
• DOKUWIKI=stable
• DOKUWIKI=old-stable before_install:

Dave has gone back to PHP 5.6 (which shows errors codes in our Tracker - but tracker works!) because we cannot set PHP version 5.6 on separate directories, and my installation of the selected 5.3 Siteexport version into the most current version of Dokuwiki fails with a blank screen, thus I have to delete the plugin/siteexport folder.

So, we are using the current version of SiteExport now with PHP 5.6.

[rgleason]

Gerry, Dave has set the PHP for the entire website to 5.6 so we are using the new versions of siteexport.

I have used indexmenu to create two TOC which utilize the skipns=/ feature to make the number of pages smaller. I created one TOC in the Parent Namespace and the other in the Opencpn_User_Manual namespace. They are essentially the same, but they must be different because they are located in different namespaces. Having gotten the TOC so it works right, and checked for orphans, blank pages to force pages to namespace, etc I decided to try running SiteExport from the Parent Namespace.

Set Namespace: opencpn:toc Parent Namespace to export: opencpn Specified Depth 5 Render engine: ckedit Template: dokuwiki Export linked pages checked Export all parameters checked
.hit Wget then Start

Now it says: "Adding "opencpn:developer_manual:plugins:beta_plugins:nmea_converter" ( 41 / 496 ) " Why is it doing this? That page is not included in my TOC! Also there are entirely too many pages, I have only 140 pages included in the TOC, max 150!

It failed on timeout again with this message. Adding "opencpn:developer_manual:plugins:learning:oplaydo1" ( 52 / 496 ) -- Note that I have "skipns=/ the entire namespace "developer_manual"

SiteExport is not doing what I am expecting. What misconfiguration am I doing? Why doesn't SiteExport follow the TOC directions I have set up?

Please see this thread in Dokuwiki forum. Also these links to the website. TOC in Parent Namespace: https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:toc TOC in Opencpn User Manual Namespace: https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:toc

[rgleason]

I've now tried the Opencpn User Manual namespace TOC with a similar setup.

Set Namespace: opencpn:opencpn_user_manual:toc Parent Namespace to export: opencpn:opencpn_user_manual Specified Depth 5 Render engine: ckedit Template: dokuwiki Export linked pages checked Export all parameters checked .hit Wget then Start

Now I am getting this: Adding "opencpn:opencpn_user_manual:advanced_features:network_repeater" ( 29 / 309 )

and I had skipns= that namespace! What is happening here? only have about 140 pages to export in the TOC.

Failed at Adding "opencpn:opencpn_user_manual:charts:vector_display_tab" ( 60 / 309 ) 500 Internal Server Error Request Timeout

I don"t think siteexport is using my TOC for some reason

Attached is the TOC IndexMenu-TOC-code.txt

[rgleason]

I just tried the above setup but with "Specified Depth = 3" It completed and I downloaded the sitexport.zip. I can confirm that there are namespaces in there that I had excluded (skipns) Do I have to also do the parameter skipfile as well, matching each namespace? https://www.dokuwiki.org/plugin:indexmenu

skipfile[+\	=]/regexp/	Skip files matching the regexp

I tried it and it doe not seem to work, I get an error.

[rgleason]

Is there a technique to export or "save as" the TOC after generation? If so, I would simply do that and paste it into the TOC as hard coded TOC. Then try to run SiteExport on that! Or if there is a way to import a file into SiteExport I would do that. Simply to be sure the program is using the simplest instructions and not doing any additional work.

See Indexmenu - How to export/save as a Generated TOC in dokuwiki namespace format.

[rgleason]

1. Loaded this URL: TOC User Manual
2. Then selected the "Export Page (Site Export) in the right side menu second from bottom.
3. This brings up Site Export Manager with the following settings: A. Set Namespace: opencpn:opencpn_user_manual:toc B. Parent Namespace to export: opencpn:opencpn_user_manual:toc C. Export type: This page only D. Export Linked Pages: check E. Export all parameters (e.g. "do"): check F. Render Engine: xhtml G. Template: dokuwiki H. the rest are unchecked. I. Start the Process - Hit Curl, then hit START

A single TOC page is created. Irt looks correct!

See attached. siteexport-TOC-single-page.zip

[rgleason]

Trial on same TOC page using same right button "Export Page (Site Export) Except the following change: Depth = 3 I had to use the "WGET" button then START because Curl failed.

The html TOC page was exported and each of the html pages in opencpn:opencpn_user_manual.

See attached please siteexport-TOC-used-wget-depth-3.zip

[rgleason]

With depth set to 5 it still seems to fail on Adding "opencpn:opencpn_user_manual:charts:vector_display_tab" ( 60 / 309 ) I have added this namspace to skipns= but it does not seem to make a difference.

Is it possible that there are problems in the website that are causeing these problems? If so, what do I look for? What tools will help?

[rgleason]

I recently made it to 60 out of 142 pages. I downloaded that zip altough that was not presentedl to a previous download and unzip. Unsipping that file into a previous successful download and uxzip seemed to work perfectly with all the links correct. This action expanded the documentation withouut hittting the arbitarty limits previous.

Is there a more logical way to procede to complete the work?

[gamma]

Good morning. I'll read through the thread in a few minutes ;)

[gamma]

About the TOC. It is correct, SiteExport does not recognize the indexmenu as a TOC creation variant. I think that could be incorporated.

Have you found out which of the pages is creating the export problems? Are there several or just some pages that are problematic? Are any of these pages in your indexmenu definition?

[gamma]

Ok, the problem you're facing is probably due to exporting the page with its links is like exporting only one page - which is handled internally. That means that exporting every page from the starting point will accumulate to the time it takes to export the whole thing. Using the TOC syntax that siteexport provides it can check through the pages and handle each one individually.

There are two things that I think could mitigate the problem:

1. Each page that comes from a link should be handled individually and be added to the export queue that is visible from the GUI
2. The siteexport plugin should be able to handle the indexmenu generated toc

[gamma]

Can you please check the newest version from the branch indexmenu-toc? Please make sure that you select Export all subnamespaces

When clicking the start button it should show you that it will export x/y pages.

For now I'd suggest to disable the Export linked pages option.

[rgleason]

Gerry, Thank you for yor help! I have read the suggestions above, do not completely understand them but will read them again. Today I worked on cleaning up orphans, wanted media and content. The list was quite long. Directly afterwards I tried a siteexport from the TOC page using the same parameters as before. Then I decided to check github.

Adding "opencpn:opencpn_user_manual:charts:vector_display_tab" ( 60 / 378 ) 500 Internal Server Error Request Timeout

[rgleason]

TOC

1. SiteExport does not recognize the indexmenu as a TOC creation variant. That could be incorporated.
2. No, I have not found out which pages yet. I have tried removing the pages that were exporting when failure occurred. That started to seem random, because the #of pages did not always increase.
3. I believe all of the pages are from my indexmenu definition, because I basically create the entire TOC with an indexmenu code, similar to this but much longer.
4. Problem is due to the large Indexmenu TOC which is a single page that runs out of time,- time accumulates from the start, the page does not complete until the whole document is completed..

{{indexmenu>:opencpn:opencpn_user_manual#5|msort nsort nojs skipns=/(^opencpn:opencpn_user_manual:getting_around:gps_setup_and_status$|^opencpn:opencpn_user_manual:charts:advanced_chart_work$|^opencpn:opencpn_user_manual:advanced_features:extended_marks$|^opencpn:opencpn_user_manual:plugins:charts:nv_charts$| .....

That is why I thought it might be helpful to be able to save the generated output TOC on dokuwiki namespace format, and then just paste it into the same TOC page over the indexmenu generating code. Then the siteexport would just read the hard code of the TOC.

Your solution:

1. Each page that comes from a link should be handled individually and be added to the export queue that is visible from the GUI
   2. The siteexport plugin should be able to handle the indexmenu generated toc

This sounds great. I am guessing that is what you've done in github below. I am going to give it a try.

[rgleason]

Gerry by the way, I've been using multiOrphan almost all day. I like it! Things should be much cleaner now. I still have a few photos that should be moved to the standard places.

[rgleason]

Dear Gerry, I've forked your recent commit and downloaded it as a zip. Is that ready to install manually? I thought rather than installing, I would try NOT to use IndexMenu TOC, and ended up finding the "Sitemap" below the "Admin". So I tried a number of times with Sitemap, then pick the Export -Siteexport button from the right side of the page, then I found that ckedit and dokuwiki template worked best, Kept export linked checked generally, and then tried using the "Direct Download" button and this worked this time, so I've been using that. I got up to 81 pages done at with 3 levels. It seems to e doing the entire site (using sitemap). I can't get Level 4 working yet.

On level 4, it has failed twice at 60 pages, on Vector Display Tab. I'm using multiOrphans to look at that page, and I found one unbracketed .pdf external link.

Besides the many "blank" page to force a page into a namespace so that indexmenu will sort by numbered index, I found only three orphaned pages. Everything else looked ok.

Is there an easy way to exclude/skip this page/namespace from processing by putting a code at the top of the page, like {{skipns-siteexport}} ? similar to {{indexmenu_n>14}} to determine page order? That would make it easier debugging this stuff.

I've also found that I can combine siteexports, provided they have the right linking. So that I can "build" the full document. In fact I copied the html for the Indexmenu TOC into a directory I am trying to "build" and it works too.

Now I am trying to figure out how to get this to start processing on lower levels. It always processes starting at the top. I need to process stuff at the lower levels because it is failing at about 60-81 pages. The only way I know how to do that is by using an Indexmenu TOC in the lower namespaces. Ah ha found the answer here = [if you just want to export a branch you have to go to this folder and there enter the admin-area; you can type in the wanted namespace ether. For me it seems to be more safe to view the namespace first!]

php().ini says max_execution_time | 30 | 30 That is seconds isnt it or is it at 30 minutes? I read that you suggest at least 2 minutes.

Would it be better to make a hard coded TOC that has each and every namespace needed and not the ones that we don't want?

Also do you know what this command is for

[gamma]

Hey Rick,

Yes, try to install the forked version (indexmenu branch). I think you can exclude pages from export with a global option in the configuration. I guess it would make sense for page based or export based exclusion of pages. That is something I can look into early next week.

Thanks for the feedback on the multiorphan plugin :)

[rgleason]

Here is what I wrote in the forum > plugins https://forum.dokuwiki.org/post/59322 I made a PR using your indexmenu branch and merged it into my master. The downloaded and installed the more recent version. Now running it.

Put this indexmenu code into my TOC page {{indexmenu>:opencpn:opencpn_user_manual#5|msort nsort nojs}} --- does the siteexport expect javascript? I have put nojs .

This is the fourth time this page has failed! Adding "opencpn:opencpn_user_manual:charts:vector_display_tab" ( 30 / 140 ) Usually it is at page 60, but the total pages have gone down to the 140 I always expected, which is good.

I am going to try to skipns=/ that vector_display_tab

[gamma]

SiteExport does not expect the JS part. It uses just pages it receives from the indexmenu in the initial output.

[rgleason]

Added vector_display_tab to the Config siteexport global exclude, the indexmenu parameter skipns=/ does not work.

See this moving gif? https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:charts:vector_display_tab#vector_chart_hot_keys Would that be the cause of the timeout? . Boy it screams through simple text pages!

Its doing better now, 144 of 307 still going.

Adding "opencpn:opencpn_user_manual:getting_around:gps_setup_and_status" ( 144 / 307 )

Will add gps_setup_and_status to global excudes.

[gamma]

There is something weird about this page. It takes quite long for it to load. I tried the following - but I do not have access to the log, so please can you check it and send me the log - export only this one page. Also check 'Disable cache for current request'.

It'd be good to clear the log file before starting so we have only the latest export information there. With log level debug enabled there should be something in the log that can help us.

[rgleason]

Almost made it. This page is big with lots of images. Adding "opencpn:opencpn_user_manual:options_setting:connections" ( 191 / 307 ) 500 Internal Server Error

Getting late. Will do in the morning.
Oh I see your post above. Will get those for you.

[rgleason]

I checked Disable Cache for current request, and tried very many settings, for render engine and templace. also turned off linking and tried each of the save button types.
It always fails. Tomorrow morning I will process that moving gif file so it does not run showing the text on and off!

[rgleason]

Gerry I am going to give send you access to the website again, but with a new password. Then I'll do the log and clear it.

[rgleason]

Sent you new a PW for your admin access. Here are the error and debug. I recently turned debug off to see if that made a difference, so it does not have the last actions.

siteexport-reduce.zip error_log.zip

Thank you!

[rgleason]

This morning, now following the instructions: (:-)

Newest version from the branch indexmenu-toc? yes Bring up the TOC User Manual page which has the generator code:

• [[opencpn:opencpn_user_manual]]
• {{indexmenu>:opencpn:opencpn_user_manual#5|msort nsort nojs}}

Select from the pages right menu, second from the bottom Export SiteExport Set Namespace= opencpn:opencpn_user_manual:toc Parent Namespace to Export= opencpn:opencpn_user_manual:toc Select "Export all subnamespaces" yes For now, Disable the "Export linked pages" option. unchecked, yes Export all parameters "do" -checked Render Engine: ckedit Template: dokuwiki Everything else is unchecked. Just hit "Start" ( without first hitting one of the three "Direct", "Wget" or "Curl") yes When clicking the start button it should show you that it will export x/y pages. yes Starts out:Adding "opencpn:opencpn_user_manual:advanced_features:maximise_screen_display" ( 27 / 379 )

• Seems faster. Now on Adding "opencpn:opencpn_user_manual:options_setting:display" ( 202 / 379 ) still running! Made it to here:Adding "opencpn:opencpn_user_manual:plugins:charts:vfkaps" ( 215 / 379 ) timeout. I am going to remove that page in the config siteexport skip area using "charts:vfkaps"

  Second try this morning: Start at 10:58 am - 11:24 am = 26 min Adding "opencpn:opencpn_user_manual:plugins:included_plugins:grib_weather" ( 224 / 379 ) timeout failure.

I am going to have to get Dave to change php max_execution_time | 30 | 30 to 45 minutes at least even though that was not the cause of the timeout failure, it looks like the whole process is going to take more than 30 minutes now and the next time it will fail because of that php parameter.

Also I will remove that page that failed in the config siteexport skip field using "included_plugins:grib_weather"

[rgleason]

Here is what I have in the dokuwiki admin > configuration under siteexport now:

plugins>siteexport>exclude

"included_plugins:grib_weather charts:vfkaps options_setting:connections gps_setup_and_status vector_display_tab .pdf zip 7z edit_user_manual blank blank.txt cruisersforum github"

[rgleason]

I started doing another export again, and this time everyting is the same, but I unchecked "Export all parameters "do" so that is disabled now.

Right now it is processing, we started at about 11:45 am "Adding "opencpn:opencpn_user_manual:advanced_features:route_to_autopilot" ( 34 / 379 )"

Failure at 12:14 pm Adding "opencpn:opencpn_user_manual:plugins:navigation:celestial_navigation" ( 251 / 379 ) 500 Internal Server Error

This is very close to the 30 minute limit in the server. Just checked it is still max_execution_time | 30 | 30

The total number of pages we are processing did not change so the excludes are applied after this value is calculated or possibly these "exclude" notations are not working (but they do seem to help with increasing numbers of pages being processed each time.

[rgleason]

plugins>siteexport>exclude

"navigation:celestial_navigation included_plugins:grib_weather charts:vfkaps options_setting:connections gps_setup_and_status vector_display_tab .pdf zip 7z edit_user_manual blank blank.txt cruisersforum github"

[rgleason]

Gerry ANOTHER IDEA: When the processing of

1. A page takes longer than 5 minutes skip it and add it to a list called AutoExclude__ located where?
2. The document please skip the pages/namespaces listed in _Exclude___ located where? User needs easy access to this list. Also the format used for defining the list should be clear. A. Dokuwiki namespaces B. If a namespace is excluded that will include all namespaces below by default. B. If an namespace is listed under excluded but asked to be "blank" then don't process this page, just leave it "Blank" for now and include the namespace and process all namespaces below?

[rgleason]

After excluding celestial_navigation started another export. Now within 100 pages! Adding "opencpn:opencpn_user_manual:plugins:radar_overlay_ais:br24_radar" ( 270 / 379 )

Whoops spoke too soon took about 28 minutes. Adding "opencpn:opencpn_user_manual:plugins:radar_overlay_ais:br24_radar" ( 270 / 379 ) 500 Internal Server Error Request Timeout

Will exclude this too. radar_overlay_ais:br24_radar

Now starting over at 1:02pm Don't know when it stopped.

It just hiccupped on the page 33 which it processed before at least a dozen times! Adding "opencpn:opencpn_user_manual:advanced_features:portable_opencpn_v2" ( 33 / 379 )

I'll add this one too.

[rgleason]

Excluded advanced_features:portable_opencpn_v2 and restarted at 1:29 pm Adding "opencpn:opencpn_user_manual:plugins:safety:odraw1.4_pi" ( 276 / 379 ) 500 Internal Server Error

IT WOULD HELP to have the time of the Internal Server Error, as I have other things going on too.

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:safety:sar" ( 277 / 379 )

Started a now one 3:18 pm

Adding "opencpn:opencpn_user_manual:plugins:safety:watchdog" ( 278 / 379 ) Since it is now 3:45 pm and I am now removing sequential files. I believe the problem is the php max time of 30 minutes, so I am not going to remove this one. Bdbcat needs to give me another 20 minutes or so.

[rgleason]

Gerry Any ideas for me? I seem to be removing sequential files and they appear to be at the PHP max time of 30 minutes.

I am trying to get Dave to extend that value before continuing. Do you think it matters?

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:safety:watchdog" ( 278 / 379 ) a little better. 10:48 pm

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:sailing_tools:tactics" ( 284 / 379 ) 500 Internal Server Error 11:11 pm

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:utility_plugins:launcher" ( 290 / 379 ) 11:36 pm

[rgleason]

"opencpn:opencpn_user_manual:plugins:utility_plugins:launcher" ( 290 / 379 ) I thought I excluded that!! 12:12pm ... I did not.

[rgleason]

max_execute_time extended to 45 minutes. Now a page will not process, but we wait until 45 min to get an error. Adding "opencpn:opencpn_user_manual:plugins:charts:bsb4_charts" ( 211 / 379 ) no timeout error but hung up, excluded.

[rgleason]

Turned off all plugins to speed up processing.

Adding "opencpn:opencpn_user_manual:plugins:utility_plugins:object_search" ( 291 / 379 ) Time out error --excluding.

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:weather:climatology" ( 298 / 379 ) Time out error - excluding

[rgleason]

Trying to run two separate machines and processes. This seems to be going backwards, because it passed before, many times, but I will exclude it after the other machine has finished. Adding "opencpn:opencpn_user_manual:plugins:charts:nv_charts" ( 212 / 379 )

[rgleason]

Adding "opencpn:opencpn_user_manual:plugins:weather:weather_routing" ( 300 / 379 ) timee out error will exclude. Yikes this is going to end up excluding all 70 pages at this rate.