Laypa
Laypa: A Novel Framework for Applying Segmentation Networks to Historical Documents
HIP'23 paper: https://doi.org/10.1145/3604951.3605520
ArXiv paper: Coming soon!
Part of the [Loghi pipeline][loghi_link]
Laypa is a segmentation network, with the goal of finding regions (paragraph, page number, etc.) and baselines in documents. The current approach is using a ResNet backbone and a feature pyramid head, which made pixel wise classifications. The models are built using the [detectron2][detectron_link] framework. The baselines and region classifications are then made available for further processing. This post-processing turn the classification into instances. So that they can be used by other programs (OCR/HTR), either as masks or directly as pageXML.
Table of Contents
Tested Environments
Developed using the following software and hardware:
| Operating System | Python | PyTorch | Cudatoolkit | GPU | CPU | Success |
|---|---|---|---|---|---|---|
| Ubuntu 22.04.4 LTS (Linux-6.5.0-28-generic-x86_64-with-glibc2.35) | 3.12.3 | 2.3.0 | 12.1 | NVIDIA GeForce RTX 3080 Ti Laptop GPU | 12th Gen Intel(R) Core(TM) i9-12900H | :white_check_mark: |
Click here to show all tested environments
More coming soon Run [`utils/collect_env_info.py`][collect_env_info_link] to retrieve your environment information, and add them via [pull request][pull_request_link]. | Operating System | Python | PyTorch | Cudatoolkit | GPU | CPU | Success | | ----------------------------------------------------------------- | ------ | ------- | ----------- | ------------------------------------- | ------------------------------------ | ------------------ | | Ubuntu 22.04.4 LTS (Linux-6.5.0-28-generic-x86_64-with-glibc2.35) | 3.12.3 | 2.3.0 | 12.1 | NVIDIA GeForce RTX 3080 Ti Laptop GPU | 12th Gen Intel(R) Core(TM) i9-12900H | :white_check_mark: |Setup
The recommended way of running Laypa is inside a conda environment. To ensure easier compatibility a method of building a docker is also provided.
To start clone the github repo to your local machine using either HTTPS:
git clone https://github.com/stefanklut/laypa.gitOr using SSH:
git clone git@github.com:stefanklut/laypa.gitAnd make laypa the working directory:
cd laypaConda
If not already installed, install either conda or miniconda ([install instructions][conda_install_link]), or mamba ([install instructions][mamba_install_link]).
The required packages are listed in the [environment.yml][environment_link] file. The environment can be automatically created using the following commands.
Using conda/miniconda:
conda env create -f environment.ymlUsing mamba:
mamba env create -f environment.ymlWhen running Laypa always activate the conda environment
conda activate laypaDocker
If not already installed, install the Docker Engine ([install instructions][docker_install_link]). The docker environment can most easily be build with the provided script.
Download from dockerhub
Laypa now has a release on dockerhub. Using the docker of loghi/docker.laypa, should pull the corresponding laypa docker directly from docker hub. If this fails from some reason it can be pulled manually from [here][dockerhub_link]. If it is outdated or requires differences to the source code, please try the Manual Installation.
Manual Installation
Building the docker using the provided script:
./buildImage.sh <PATH_TO_LAYPA>Or the multistage build with some profiler tools taken out (might be smaller):
./buildImage.multistage.sh <PATH_TO_LAYPA>Click for manual docker install instructions (not recommended)
First copy the Laypa directory to the temporary docker directory: ```sh tmp_dir=$(mktemp -d) cp -r -TClick for minikube install instructions
Minikube is local Kubernetes, allowing you to test the Laypa tools in a Kubernetes environment. If not already installed start with installing minikube ([install instructions][minikube_install_link]) If the docker images have already been built the minikube can run them straight away. To do so, start minikube without any special arguments: ```sh minikube start ``` Afterwards the docker for Laypa can be added to the running minikube instance using the following command (assuming the Laypa docker was built under the name loghi/docker.laypa): ```sh minikube image load loghi/docker.laypa ``` It is also possible to build the Laypa docker using the minikube docker instance. This means minikube will need access to the Laypa code. As it stand, this is current still done using a copy command from the local storage. In order to do so start the minikube with the mount argument: ```sh minikube start --mount ``` This will make the machines filesystem available to minikube. Then ssh into the running minikube: ```sh minikube ssh ``` Within the ssh minikube go to the location of the laypa where the host `/home/When successful the docker image should be available under the name loghi/docker.laypa. This can be verified using the following command:
docker image lsAnd checking if loghi/docker.laypa is present in the list of built images.
Pretrained models
Some initial pretrained models can be found [here][pretrained_models_link].
Dataset(s)
The dataset used for training requires images combined with ground truth pageXML. For structure the pageXML needs to be inside a directory one level down from the images. The dataset can be split over multiple directories, with the image paths specified in a .txt file. The structure should look as follows:
training_data
├── page
│ ├── image1.xml
│ ├── image2.xml
│ ├── image3.xml
│ └── ...
├── image1.jpg
├── image2.jpg
├── image3.jpg
└── ...Where the image and pageXML filename stems should match image1.jpg <-> image1.xml. For the .txt based dataset absolute paths to the images are recommended. The structure for the data used as validation is the same as that for training.
When running inference the images you want processed should be in a single directory. With the images directly under the root folder as follows:
inference_data
├── image1.jpg
├── image2.jpg
├── image3.jpg
└── ...Some dataset that should work with laypa are listed below, some preprocessing may be require:
- [cBAD][cbad_link]
- [VOC and notarial deeds][voc_link]
- [OHG][ohg_link]
- [Bozen][bozen_link]
Training
Three things are required to train a model using [main.py][main_link].
- A config file, See [
configs/segmentation][configs_link] for examples of config files and their contents. - Ground truth training/validation data in the form of images and their corresponding pageXML. The training/validation data can be provided by giving either a
.txtfile containing image paths, the image paths themselves, or the path of a directory containing the images.
Required arguments:
python main.py \
-c/--config <CONFIG> \
-t/--train <TRAIN [TRAIN ...]> \
-v/--val <VAL [VAL ...]>Click to see all arguments
Optional arguments: ```sh python main.py \ -c/--config CONFIG \ -t/--train TRAIN [TRAIN ...] \ -v/--val VAL [VAL ...] \ [--tmp_dir TMP_DIR] \ [--keep_tmp_dir] \ [--num-gpus NUM_GPUS] \ [--num-machines NUM_MACHINES] \ [--machine-rank MACHINE_RANK] \ [--dist-url DIST_URL] \ [--opts OPTS [OPTS ...]] ``` The optional arguments are shown using square brackets. The `--tmp_dir` parameter specifies a folder in which to store temporary files. While the `--keep_tmp_dir` parameter prevents the temporary files from being deleted after a run (mostly for debugging). The remaining arguments are all for training with multiple GPUs or on multiple nodes. `--num-gpus` specifies the number of GPUs per machine. `--num-machines` specifies the number of nodes in the network. `--machine-rank` gives a node a unique number. `--dist-url` is the URL for the PyTorch distributed backend. The final parameter `--opts` allows you to change values specified in the config files. For example, `--opts SOLVER.IMS_PER_BATCH 8` sets the batch size to 8.As indicated by the trailing dots multiple training sets can be passed to the training model at once. This can also be done using the train argument multiple types. The .txt files can also be mixed with the directories. For example:
# Pass multiple directories at once
python main.py -c config.yml -t data/training_dir1 data/training_dir2 -v data/validation_set
# Pass multiple directories with multiple arguments
python main.py -c config.yml -t data/training_dir1 -t data/training_dir2 -v data/validation_set
# Mix training directory with txt file
python main.py -c config.yml -t data/training_dir -t data/training_file.txt -v data/validation_setInference
To run the trained model on images without ground truth, the images need to be in a single directory. The output consists of either pageXML in the case of regions or a mask in the other cases. This mask can then be processed using other tools to turn the pixel predictions into valid pageXML (for example on baselines). As stated, the regions are turned into polygons for the pageXML within the program already.
How to run the Laypa inference individually will be explained first, and how to run it with the full scripts that include the conversion from images to pageXML with come after.
Without External Processing
To just run the Laypa inference in [run.py][run_link], you need three things:
- A config file, See [
configs/segmentation][configs_link] for examples of config files and their contents. - The data can be provided by giving either a
.txtfile containing image paths, the image paths themselves, or the path of a directory containing the images. - A location to which the processed files can be written. The directory will be created if it does not exist yet.
Required arguments
python run.py \
-c/--config CONFIG \
-i/--input INPUT \
-o/--output OUTPUTClick to see all arguments
Optional arguments: ```sh python run.py \ -c/--config CONFIG \ -i/--input INPUT \ -o/--output OUTPUT [--opts OPTS [OPTS ...]] ``` The optional arguments are shown using square brackets. The final parameter `--opts` allows you to change values specified in the config files. For example, `--opts SOLVER.IMS_PER_BATCH 8` sets the batch size to 8. List values have to be overridden by encapsulating the whole list with quotes like `--opts PREPROCESS.REGION.RECTANGLE_REGIONS '["Photo"]'`An example of how to call the run.py command is given below:
python run.py -c config.yml -i data/inference_dir -o results_dirWith External Java Processing
Examples of running the full pipeline (with processing of baselines) are present in the [scripts][scripts_link] directory. These files make the assumption that the docker images for both Laypa and the loghi-tooling (Java post-processing) are available on your machine. The script will also try and verify this. The Laypa docker image needs to be build with the pretrained models included.
To run the scripts only two thing are needed:
- A directory with images to be processed.
- A location to which the processed files can be written. The directory will be created if it does not exist yet.
Required arguments:
./scripts/pipeline.sh <input> <output>Click to see all arguments
Optional arguments: ```sh ./scripts/pipeline.sh \ \The positional arguments input and output refer to the input and output directory. An example of running the one of the pipelines is shown below:
./scripts/pipeline.sh inference_dir results_dirFlask Server
The Flask Server is set up to run the inference code in a Kubernetes environment. To run the Flask API run the [start_flask.sh][start_flask_link] application with the environment variables set. This can generally be set when running a docker, which can set the environment variables beforehand depending on the docker internal file structure. To quickly test locally you can run the [start_flask_local.sh][start_flask_local_link] application, which sets the environment variables at runtime.
The flask server will run on port 5000 and can be called from outside using a curl command. When testing on a localhost the command will look as follows:
curl -X POST -F image=@<PATH_TO_IMAGE> -F identifier=<identifier> -F model=<MODEL_FOLDER_NAME> 'http://localhost:5000/predict'The required form information is the image (image) that should be processed. A given identifier to differentiate multiple runs/tests (identifier). And finally which config and weights to use (model). The config and weights are saved in a folder, this folder name is what needs to be provided. In this folder, the config should be named config.yml and the weight file should end in .pth.
Tutorial
For a small tutorial using some concrete examples see the [tutorial][tutorial_link] directory.
Evaluation
The Laypa repository also contains a few tools used to evaluate the results generated by the model.
The first tool is a visual comparison between the predictions of the model and the ground truth. This is done as an overlay of the classes over the original image. The overlay class names and colors are taken from the dataset catalog. The tool to do this is [visualization.py][eval_link]. The visualization has almost the same arguments as the training command ([main.py][main_link]).
Required arguments:
python tooling/visualization.py \
-c/--config CONFIG \
-i/--input INPUT [INPUT ...] \Click to see all arguments
Optional arguments: ```sh python tooling/visualization.py \ -c/--config CONFIG \ -i/--input INPUT [INPUT ...] \ [-o/--output OUTPUT] \ [--tmp_dir TMP_DIR] \ [--keep_tmp_dir] [--opts OPTS [OPTS ...]] \ [--sorted] \ [--save SAVE] ``` The optional arguments are shown using square brackets. The `-o/output` parameter specifies the output directory for the visualization masks. The `--tmp_dir` parameter specifies a folder in which to store temporary files. While the `--keep_tmp_dir` parameter prevents the temporary files from being deleted after a run (mostly for debugging). The final parameter `--opts` allows you to change values specified in the config files. For example, `--opts SOLVER.IMS_PER_BATCH 8` sets the batch size to 8. The `--sorted` parameter sorts the images based on the order in the operating system. The `--save` parameter specifies what type of file the visualization should be saved as. The options are "pred" for the prediction, "gt" for the ground truth, "both" for both the prediction and the ground truth and "all" for all of the previous. If just `--save` is given the default is "all".Example of running [visualization.py][eval_link]:
python tooling/visualization.py -c config.yml -i input_dirThe [visualization.py][eval_link] will then open a window with both the prediction and the ground truth side by side (if the ground truth exists). Allowing for easier comparison. The visualization masks are created in the same way the preprocessing converts pageXML to masks.
The second tool [validation.py][validation_link] is used to get the validation scores of a model. This is done by comparing the prediction of the model to the ground truth. The validation scores are the Intersection over Union (IoU) and Accuracy (Acc) scores. The tool requires the input directory (--input) where there is also a page folder inside the input folder. The page folder should contain the xmls with the ground truth baselines/regions. To run the validation tool use the following command:
Required arguments:
python tooling/validation.py \
-c/--config CONFIG \
-i/--input INPUT