check_inventory

[funderburkjim]

This issue for discussion of the check_inventory subfolder. Please read the introductory readme by clicking on the link.

[funderburkjim]

DIfference summary

webinventory.txt identifies the differences between the files managed by csl-websanlexicon (v00/inventory.txt) and the actual files in the web directory for each dictionary.

Here is a summary of the web files that are unmanaged by csl-websanlexicon, organized by folder.

• web
  • index.php -- links to (a) Available displays, (b) Downloads, (c) Related material. Unmanaged.
  • XXXheader.xml -- Credits and license. Source: pywork
• web/fonts Contains font files used for web-fonts. All dictionaries except VCP show the same list of 9 fonts (VCP omits praja).
  • this directory should be managed by csl-websanlexicon
  • The 'praja font should be removed -- it is proprietary
    • Using csl-websanlexicon to remove a file requires a new skill for generate.py.
  • Some other font files should also be removed, as they are unused, such as sanskrit2003.
• web/pdfpages Not managed by anything now.
  • MWScan/2014 uses a symlink.
  • VCPScan/2020 uses a symlink.
• web/sqlite
  • XXX.sqlite -- the sqlite3 database version of pywork/XXX.xml.
    • managed by pywork, but using components managed by csl-websanlexicon. These components are: def.sql, make_input.php, redo.sh
  • input.txt -- An transient intermediate file between XXX.xml and XXX.sqlite.
    Should be deleted.
  • XXXab.sqlite This is present in all dictionaries. BUT it is empty in most.
    • Managed manually by files in pywork (such as in directory mwab).
    • Empty XXXab.sqlite files should be removed. Not sure which dictionaries actually have 'ab'
  • mw also has mwauthtooltips.sqlite and mwkeys.sqlite. Also deletable folder 'old'
  • pwg also has pwgbib.sqlite
  • pw also has pwbib.sqlite
• web/webtc File pdffiles.txt (filenames for scanned images). Not managed anywhere.
• web/webtc2 query_dump.txt Managed by pywork/redo_xml.sh, using program init_query.py which is managed by csl-websanlexicon.
• web/webtc1 (All files managed by csl-websanlexicon)
• web/mobile1 (All files managed by csl-websanlexicon)
• web/images (All files managed by csl-websanlexicon)
• web/js (All files managed by csl-websanlexicon)

[gasyoun]

Also deletable folder 'old'

Guess most of them are. They create more mess than solve as a backup solution.

[funderburkjim]

Revised webinventory

After various file deletions, etc., the comparison is simpler between

• the files managed by csl-websanlexicon inventory.txt ,and
• the actual files within the web directory of the various dictionaries.

In addition to the files from csl-websanlexicon,  
the web directory for each dictionary has several additional required files.

Each of these files is 'dictionary specific' (i.e, NOT managed by csl-websanlexicon).

There are 5 files which are present for each dictionary.

web/<dict>header.xml
web/sqlite/<dict>.sqlite
web/index.php
web/webtc/pdffiles.txt
web/webtc2/query_dump.txt

The Cologne server also has a web/pdfpages/ directory of scanned images. As mentioned elsewhere,
servepdf now handles this more flexibly. 

There are 7 dictionaries with additional files:
ben: web/images/vatsa.png
cae: web/sqlite/caeab.sqlite
pw : web/sqlite/pwab.sqlite , web/sqlite/pwbib.sqlite
bur: web/sqlite/burab.sqlite
stc: web/index_fr.php, web/webtc/download_fr.html, web/webtc/help_fr.html , web/webtc/indexcaller_fr.php
pwg: web/sqlite/pwgab.sqlite , web/sqlite/pwgbib.sqlite
mw : web/sqlite/mwab.sqlite , web/sqlite/mwauthtooltips.sqlite , web/sqlite/mwkeys.sqlite

Implication

In order to recreate the web directory for a given dictionary X, we currently require:

• all the inventory files (of type C, T) managed by csl-websanlexicon
• all the additional files mentioned above.

[funderburkjim]

Management of non csl-websanlexicon files

The current way the additional files are managed is various, and needs revision. Before revising, here is how things are done now.

managed by pywork/redo_xml.sh

• web/sqlite/XXX.sqlite .
  • csl-websanlexicon currently manages 3 files: sqlite/def.sql, sqlite/make_input.php, and sqlite/redo.sh. redo_xml.sh changes to the web/sqlite directory and runs redo.sh.
• web/webtc2/query_dump.txt.
  • csl-websanlexicon currently manages 2 files: web/webtc2/init_query.py and web/webtc2/init_query.sh. redo_xml.sh changes to the web/webtc2 directory and runs init_query.sh.

files managed manually.

These files just are. They are managed by editing. They are not managed by csl-websanlexicon. There is one of each of these files for each dictionary.

• web/XXXheader.xml
• web/index.php
• web/webtc/pdffiles.txt

misc files managed manually

These files are peculiar to given dictionaries. They are not managed by csl-websanlexicon

• ben: web/images/vatsa.png (referenced by digitization of entry vatsa.
  • There may be entries in other dictionaries where the scanned image uses an image. We could follow the model of Benfey in vatsa to use images in those cases.
• stc: stc: web/index_fr.php, web/webtc/download_fr.html, web/webtc/help_fr.html , web/webtc/indexcaller_fr.php
  These files were developed with O. Caujolle to provide internationalization for some of the STC displays. There also are English versions (which managed by csl-websanlexicon).

Abbreviations managed manually with pywork scripts

The abbreviation databases are managed within subdirectories of pywork; the resulting sqlite files are created and copied to the web/sqlite/ directory by scripts within the pywork subdirectories. However, the invocation of these scripts is manual (i.e., done only when some change occurs).

• bur: web/sqlite/burab.sqlite managed by pywork/pwab/redo.sh
• cae: web/sqlite/caeab.sqlite managed by pywork/caeab/redo.sh
• mw : web/sqlite/mwab.sqlite managed by pywork/mwab/redo.sh
• pw: web/sqlite/pwab.sqlite managed by pywork/pwab/redo.sh
• pwg: web/sqlite/pwgab.sqlite managed by pywork/pwgab/redo.sh

Literary source names managed manually with pywork scripts

The literary source abbreviation databases are managed within subdirectories of pywork; the resulting sqlite files are created and copied to the web/sqlite/ directory by scripts within the pywork subdirectories. However, the invocation of these scripts is manual (i.e., done only when some change occurs).

• mw: web/sqlite/mwauthtooltips.sqlite managed by pywork/mwauth/redo.sh
• pw: web/sqlite/pwbib.sqlite managed by pywork/pwauth/redo.sh
• pwg: web/sqlite/pwgbib.sqlite managed by pywork/pwgauth/redo.sh

mwkeys.sqlite managed manually with pywork script

Note: The reason for mwkeys.sqlite is obscure, but I think required -- something to do with weeding out the HxA (duplicate) headwords. I know one of the intermediate steps in constructing mwkeys.sqlite is used by sanhw1 construction (namely scans/awork/sanhw1/pywork/mwkeys/extract_keys_b.txt).

• mw: web/sqlite/mwkeys.sqlite managed by pywork/mwkeys/redo.sh.
  • This script is run by script redo_hw.sh, which is part of the update process when a change is made to the mw digitization.

[funderburkjim]

Rationalizing the management of entire web directory

In light of the above fairly thorough survey, some simplifications come to mind. These will be developed in other issues. We are getting closer to a minimal viable structure for 2020 version of the dictionary subdirectories.