Store link tracker dict in link_tracker.json and use later + CLI arg parsing improvements

[robintw]

This PR includes two main changes:

• Storing the link tracker (ie. 'shopping list') in a JSON file called link_tracker.json and use that when processing single files using the parse_single_file.py script so we don't have to run Run 1 in that case
• Extending the command-line arguments that can be used on the main parser script. The new --help output is below:

usage: lman_parser.py [OPTION] DATA_PATH [LOGGING_LEVEL]

positional arguments:
  DATA_PATH             Path to the source data
  LOGGING_LEVEL         Debug level - must be one of debug, warning or info

options:
  -h, --help            show this help message and exit
  --skip-first-run, --no-skip-first-run
                        Skip Run 1, loading the shopping list from the link_tracker.json file
  --warn-on-blank-runs, --no-warn-on-blank-runs
                        Print warning messages whenever runs of blank paragraphs are found

This includes two new options:

• --skip-first-run will skip doing Run 1, and use the data from link_tracker.json instead
• --warn-on-blank-runs will show warnings for situations where there are runs of

  tags just containing blank space

Both of these options are off by default, and are turned on by providing that command-line switch.

This PR makes running for single files work properly for the first time (we don't have to use nasty hacks now we're using the link_tracker.json file), and should enable clearing up the output (by removing blank tags warnings unless we turn them on) and speedier iteration (by skipping the first run when requested).

Happy to change the name of any of the CLI params if you want - just let me know.

[robintw]

Fixes #557

[IanMayo]

Thanks @robintw . So, --skip-first-run controls the behaviour of lman-parser, but the single file converter always skips first run, right?

[robintw]

Did you mean to close this @IanMayo - or did you mean to merge it?

[robintw]

And yes, --skip-first-run controls the behaviour of the main parser. Skipping the first run is always done in the single file parser (the content tester is completely separate and doesn't do any of these runs)

[IanMayo]

Did you mean to close this @IanMayo - or did you mean to merge it?

oops. Merging...

[IanMayo]

(the content tester is completely separate and doesn't do any of these runs)

Oops, I meant to refer to the single file converter, not the content tester.

[IanMayo]

Can I ask for one small extension please @robintw ?

Could we introduce a --skip-validation step?

The validation step hasn't reported errors for a couple of weeks, I don't need to run it every time.

[robintw]

Would it be better to have a --run-validation argument then? So by default it doesn't run, but you can tell it to run the validation?

[IanMayo]

Would it be better to have a --run-validation argument then? So by default it doesn't run, but you can tell it to run the validation?

(sorry for slow reply)

Yes, a logical strategy. Thanks!

[robintw]

Done.

New CLI help text is:

usage: lman_parser.py [OPTION] DATA_PATH [LOGGING_LEVEL]

positional arguments:
  DATA_PATH             Path to the source data
  LOGGING_LEVEL         Debug level - must be one of debug, warning or info

options:
  -h, --help            show this help message and exit
  --skip-first-run, --no-skip-first-run
                        Skip Run 1, loading the shopping list from the link_tracker.json file
  --warn-on-blank-runs, --no-warn-on-blank-runs
                        Print warning messages whenever runs of blank paragraphs are found
  --run-validation, --no-run-validation
                        Run the DITA validation step

[IanMayo]

It's great Robin. The single file parser runs in a flash now :-)

I'll merge it, so I can start testing it onsite shortly - but if/when you get chance could you also update the README in repo root?

That's my "go-to" stop for reminding myself of the command structure.