DBIAIT project for C179
Version is 3.16.0 Hannover
WARNING: Importing QGis forces the use of the default, global python interpreter.
Version is 3.6.9
PYTHONUNBUFFERED=1; OSGEO4W_ROOT=C:\OSGeo4W64; PATH=%OSGEO4W_ROOT%\apps\qgis\bin\;%OSGEO4W_ROOT%\apps\Python37\;%OSGEO4W_ROOT%\apps\Python37\Scripts\;%OSGEO4W_ROOT%\apps\qt5\bin\;%OSGEO4W_ROOT%\apps\Python37\Scripts\;%OSGEO4W_ROOT%\bin\;C:\Windows\;C:\Windows\system32\WBem\;%OSGEO4W_ROOT%\apps\qgis\python\plugins; PYTHONHOME=%OSGEO4W_ROOT%\apps\Python37; QGIS_PREFIX_PATH=%OSGEO4W_ROOT%\apps\qgis; QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins\;%OSGEO4W_ROOT%\apps\qt5\plugins; PYTHONPATH=%OSGEO4W_ROOT%\apps\qgis\python
export PYTHONPATH=/usr/share/qgis/python/:/usr/share/qgis/python/plugins:/usr/lib/python3/dist-packages/qgis:/usr/share/qgis/python/qgis
database/install.sql
on the databasedatabase/functions.sql
on the databasenote: if some errors occurs with windows installation of the python-ldap
, please install the library via whl file and then re-run the installation of the dependencies
This section contains information about the folder used by the authorized users (via FTP) to configure the DBIAIT application in terms of import/export options and geopackages
FTP_FOLDER: folder representing the root of the FTP for the users
IMPORT_FOLDER: folder containing geopackages and the subfolder config with the configuration files for the import process:
EXPORT_FOLDER: folder containing generated *.zip archives
EXPORT_CONF_DIR: (default: EXPORT_FOLDER/config
) folder containing configuration subdirectories for the 'Analysis' and 'Freeze' schemas export.
Configuration for 'Analysis' schema is taken from EXPORT_CONF_DIR/current/
directory and configurations for 'Freeze'
schemas are taken from EXPORT_CONF_DIR/<year>/
directories.
Each configuration subdirectory needs to define the following:
/srv/ftp
└───import
| └───config
| | | domains.csv
| | | layers.json
| | *.gpkg
└───export
| └───config
| | └───current
| | | └───sheet_configs
| | | | | <sheet_1_config>.json
| | | | | ...
| | | | | <sheet_n_config>.json
| | | | xls.json
| | | | shp.json
| | | | NETSIC_SEED.xlsx
# not yet supported, prepared for FREEZE
| | └───<year>
| | | | xls.json
| | | | shp.json
| | | | NETSIC_SEED.xlsx
QGIS_PATH: path to the QGIS application
DATABASE_HOST: PostgreSQL host
DATABASE_PORT: PostgreSQL port (default is 5432)
DATABASE_NAME: name of the database
DATABASE_USER: username to connect to the database
DATABASE_PASSWORD: password to connect to the database
The scheduler requires a task queue to be configured. By default RabbitMQ is used, but Redis can also be used (note: Redis requires change in settings.py file).
docker run -d -p 15672:15672 -p 5672:5672 -p 5671:5671 --hostname my-rabbitmq rabbitmq:3.6.10-management
python manage.py global_interpreter_rundramatiq
python manage.py runserver 0.0.0.0:8000
Note: On every change of your Dramatiq tasks it is required to restart dramatiq
process, otherwise your changes may not be included in the next background task execution
Background Task execution should be ordered similarly to Dramatiq's GenericActor
, with the only difference that Task.send() method takes a single argument - ORM Task ID, which is created by pre_send() class method.
Task.send(Task.pre_send(*args, **kwargs))
In case ordering Task execution violates querying criteria, the app.scheduler.exceptions.QueuingCriteriaViolated
will be raised.
All tasks should inherit from app.scheduler.tasks.base_task.BaseTask
class, and they should always define the following methods and attributes:
PROCESS
, IMPORT
, FREEZE
, EXPORT
) scheduler.exceptions.QueuingCriteriaViolated
.task_id
) along with args and kwargs defined in the ORM Task instance, which was created by pre_send()
method. There are two possibilities to manually start export task, but they both require django environment:
from app.scheduler.tasks import ExportTask
from django.contrib.auth import get_user_model
user = get_user_model().objects.get(pk=
2. asynchronously (with Dramatiq)
``` python
from app.scheduler.tasks import ExportTask
from django.contrib.auth import get_user_model
user = get_user_model().objects.get(pk=<requesting_user_id>)
ExportTask.send(ExportTask.pre_send(user))
Warning: Please note that a simple change of method will result in sync/async execution
Warning: Production trigger should never execute export synchronously!
To start the export process, it is required to set the EXPORT_CONF_DIR environment variable, which points at a directory meeting criteria of Application configuration.
For 'analysis' schema export configuration is taken from current
subdirectory of EXPORT_CONF_DIR, and
for 'freeze' schema export is taken from <year>
subdirectory, depending on export's year setting.
Note: In the production environment <year>
directories should not be updated manually - they will
be copied from current
when the freeze task is executed, to ensure consistency between stored historical
data and it's export configuration.
xls.json
the file defining the sheet configurations list, as a relative path ot the xls.json
file
Example xls.json
contents:
{
"xls_sheet_configs": [
"sheet_configs/config_sollev_pompe.json",
"sheet_configs/config_potabilizzatori.json",
"sheet_configs/config_sorgenti.json",
"sheet_configs/config_pozzi.json",
"sheet_configs/config_sollevamenti.json",
"sheet_configs/config_scaricatori.json",
"sheet_configs/config_pozzi_pompe.json"
]
}
sheet_configs
directory (or similar, provided in xls.json
file) containing sheet configuration
files for each Excel sheet. Sheet config files should consist of a single object, with keys defined
by the project documentation
(see documentation).
NETSIC_SEED.xlsx
seed Excel file to be used as a template for the export process, containing
defined sheets and their headers
shp.json
shapefile configuration file
To validate the configuration file along with all sheet configs, in django environment, one can use the following script:
from app.scheduler.tasks.export_definitions.config_scraper import ExportConfig
export_config = ExportConfig()
Creating an instance of ExportConfig, parses the config file and sheet config files, and validates the schema against the pattern.
In order to parse a single sheet configuration, the following may be done:
from app.scheduler.tasks.export_definitions.config_scraper import export_config_schema
import json
sheet_config_path = "<you_config_file_path>"
with open(sheet_config_path, 'r') as scp:
sheet_config = json.load(scp)
export_config_schema.validate(sheet_config)
All the tests runs under django environment.
To run them, use the following command:
Normal:
python manage.py test --settings=app.test_settings
Verbosity Mode:
python manage.py test --settings=app.test_settings -v 2