instrument scrape logic with async process steps

[WardCunningham]

This is work in progress that can be merged even though the scrape is not yet properly engaged.

Some small changes.

• denowiki.sh: added --allow-write before the file argument as is required to be effective.
• site.ts: added a console.log of generated pages for debugging generated page json.

Larger changes to scrape meta-sites

• removed metatext.localhost as the scrape meta-site now demonstrates the feature.
• removed the second version of the triple loop instrumentation to focus instead on scrape.
• changed scrape to use infinite loop instead of a relentless clock (loop is easier to pause)
• console.log metaPages to understand generated page problems

I will continue to use the simple loop just to be sure my start/stop logic is working. It seems rather brittle. The scraper didn't "just work" and failed in ways I don't understand. I will comment out lots of code and study the experience of bringing it back a little bit at a time. This way I will be developing methods for exploring code that has not yet been fully written.

[joshuabenuck]

Please add the same --allow-write flag to denowiki.cmd.

If you will be constraining your writes to a specific directory, let's pass that to allow-write (i.e. --allow-write=<dir>). I may also constrain reads in a similar manner at some point. It is much easier to gradually relax constraints (and determine how to programmatically do so step-by-step) than to add them later on once the lack of constraints causes a problem.

[WardCunningham]

I wonder if the script could be made smart enough to place the arguments in the right place on the deno command line. I'm unsure what the rules are but I could explore this. I'm thinking of some sort of grep logic based on args that we know should be early.

[WardCunningham]

Tip: remove data directory before launching:

rm -rf data; ./denowiki.sh --meta-sites-dir=./meta-sites@localtest.me

[joshuabenuck]

I wonder if the script could be made smart enough to place the arguments in the right place on the deno command line. I'm unsure what the rules are but I could explore this. I'm thinking of some sort of grep logic based on args that we know should be early.

I find myself manually running the command from denowiki.sh in order to make it easier to add more flags. One of my next changes is going to be adding some custom flags to enable or disable log messages to make debugging easier, but that technique works only because it isn't related to a permission.

Having denowiki.sh be smart enough to move permission flags to their correct location on the command line would be really helpful!

[joshuabenuck]

Thanks for the tip. The error message that is displayed when data is present doesn't give one a lot to go on.

I managed to learn how to test the changes locally before merging this time!

It's fun to see the scraper's workflow starting to come together.

[joshuabenuck]

I just noticed that the server always crashes upon startup if you ran the scraper previously. I expect you know that already and that fixing this will be part of your decomposition of the scraper logic to support incremental runs. Mentioning it just in case it wasn't already on your radar.

[WardCunningham]

I think the crash early strategy makes sense when exposing your own project to the federation. It is not so great if you have a repo with multiple people's meta-sites and you haven't setup what they need to run.

[joshuabenuck]

I mentioned in an earlier comment I was going to look into making it easier to enable / disable logging. That's proving to be a bit more work than I expected and not a yak I want to shave at the moment so I guess we'll be living with console log debugging for a bit longer.

Here are some more details for the adventurous (which may or may not end up being my future self): The log module in deno std doesn't include documentation showing how to change the global log level at runtime and the logger configured for named loggers not configured in advance is a no-op logger. To do what I want to do will require overriding the default named logger behavior and tracking down how to set the global log level (likely through some code spelunking).

https://deno.land/std/log/