search

QCDIS / NaaVRE

BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

We get missing_whitespace_around_operator #836

Closed skoulouzis closed 11 months ago

skoulouzis commented 1 year ago

When conterlizing a cell conaingn '+' with space we get:

    HTTPServerRequest(protocol='http', host='school.lifewatchdev.eu', method='POST', uri='/vl-laserfarm/user//vre/containerizer/addcell?1695371969153', version='HTTP/1.1', remote_ip='10.244.0.0')
    Traceback (most recent call last):
      File "/venv/lib/python3.9/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
      File "/venv/lib/python3.9/site-packages/jupyterlab_vre/component_containerizer/handlers.py", line 284, in post
        build_templates(cell=current_cell, files_info=files_info)
      File "/venv/lib/python3.9/site-packages/jupyterlab_vre/component_containerizer/handlers.py", line 522, in build_templates
        compiled_code = autopep8.fix_code(compiled_code)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 3506, in fix_code
        return fix_lines(sio.readlines(), options=options)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 3569, in fix_lines
        fixed_source = fix.fix()
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 613, in fix
        self._fix_source(filter_results(source=''.join(self.source),
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 557, in _fix_source
        modified_lines = fix(result)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 761, in fix_e225
        pycodestyle.missing_whitespace_around_operator(fixed, ts))
    AttributeError: module 'pycodestyle' has no attribute 'missing_whitespace_around_operator'
[W 2023-09-22 08:39:29.664 ServerApp] Unhandled error
skoulouzis commented 1 year ago

cannot reproduce. Tested with cell:

#Anti-pattern  
some_list = range(a, b+1)
msg+= 'a'
skoulouzis commented 11 months ago

We got the bug again. stacktrace:

current_cell: {
    "all_inputs": [
        "tiles"
    ],
    "base_image": "qcdis/miniconda3-pdal",
    "chart_obj": {
        "hovered": {},
        "links": {},
        "nodes": {
            "6d94954": {
                "id": "6d94954",
                "ports": {
                    "remote_path_norm": {
                        "id": "remote_path_norm",
                        "properties": {
                            "color": "#a1c587"
                        },
                        "type": "right"
                    },
                    "tiles": {
                        "id": "tiles",
                        "properties": {
                            "color": "#406abf"
                        },
                        "type": "left"
                    }
                },
                "position": {
                    "x": 35,
                    "y": 15
                },
                "properties": {
                    "deps": [
                        "",
                        "laserfarm"
                    ],
                    "inputs": [
                        "tiles"
                    ],
                    "og_node_id": "6d94954",
                    "outputs": [
                        "remote_path_norm"
                    ],
                    "params": [
                        "param_password",
                        "param_login",
                        "param_username",
                        "param_hostname"
                    ],
                    "title": "S6 Normalization-USER",
                    "vars": [
                        {
                            "color": "#406abf",
                            "direction": "input",
                            "name": "tiles",
                            "type": "datatype"
                        },
                        {
                            "color": "#a1c587",
                            "direction": "output",
                            "name": "remote_path_norm",
                            "type": "datatype"
                        }
                    ]
                },
                "type": "input-output"
            }
        },
        "offset": {
            "x": 0,
            "y": 0
        },
        "scale": 1,
        "selected": {}
    },
    "confs": {
        "conf_local_tmp": "conf_local_tmp = pathlib.Path('/tmp')",
        "conf_remote_path_norm": "conf_remote_path_norm = pathlib.Path( '/webdav/vl-laserfarm' + '/norm_'+param_username)",
        "conf_remote_path_retiled": "conf_remote_path_retiled = pathlib.Path( '/webdav/vl-laserfarm' + '/retiled_'+param_username)",
        "conf_wd_opts": "conf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}"
    },
    "container_source": "",
    "dependencies": [
        {
            "asname": null,
            "module": "laserfarm",
            "name": "DataProcessing"
        },
        {
            "asname": null,
            "module": "",
            "name": "copy"
        },
        {
            "asname": null,
            "module": "",
            "name": "pathlib"
        },
        {
            "asname": null,
            "module": "",
            "name": "json"
        }
    ],
    "inputs": [
        "tiles"
    ],
    "kernel": "ipython",
    "node_id": "6d94954",
    "original_source": "conf_local_tmp = pathlib.Path('/tmp')\nconf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}\nconf_remote_path_retiled = pathlib.Path( '/webdav/vl-laserfarm' + '/retiled_'+param_username)\nconf_remote_path_norm = pathlib.Path( '/webdav/vl-laserfarm' + '/norm_'+param_username)\n\n\ntiles\n\nremote_path_norm = str(conf_remote_path_norm)\n\nnormalization_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_retiled.as_posix(),\n    'load': {'attributes': 'all'},\n    # Filter out artifically high points - give overflow error when writing\n    'apply_filter': {'filter_type':'select_below',\n                     'attribute': 'z',\n                     'threshold': 10000.},  # remove non-physically heigh points\n    'normalize': 1,\n    'clear_cache' : {},\n    'pushremote': conf_remote_path_norm.as_posix(),\n}\n\nwith open('normalize.json', 'w') as f:\n    json.dump(normalization_input, f)\n    \n\n\nfor tile in tiles:\n    normalization_input_ = copy.deepcopy(normalization_input)\n    normalization_input_['export_point_cloud'] = {'filename': '{}.laz'.format(tile),'overwrite': True}\n    dp = DataProcessing(tile, label=tile).config(normalization_input_).setup_webdav_client(conf_wd_opts)\n    dp.run()\n\nremote_path_norm",
    "outputs": [
        "remote_path_norm"
    ],
    "params": [
        "param_password",
        "param_login",
        "param_username",
        "param_hostname"
    ],
    "task_name": "s6-normalization-USER",
    "title": "S6 Normalization-USER",
    "types": {
        "param_hostname": "str",
        "param_login": "str",
        "param_password": "str",
        "param_username": "str",
        "remote_path_norm": "str",
        "tiles": "list"
    }
}
--------------------------------------
ExtractorHandler. payload: {
    "save": false,
    "kernel": "ipython",
    "cell_index": 14,
    "notebook": {
        "metadata": {
            "kernelspec": {
                "display_name": "Python 3 (ipykernel)",
                "language": "python",
                "name": "python3"
            },
            "language_info": {
                "codemirror_mode": {
                    "name": "ipython",
                    "version": 3
                },
                "file_extension": ".py",
                "mimetype": "text/x-python",
                "name": "python",
                "nbconvert_exporter": "python",
                "pygments_lexer": "ipython3",
                "version": "3.9.16"
            }
        },
        "nbformat_minor": 5,
        "nbformat": 4,
        "cells": [
            {
                "cell_type": "code",
                "source": "import fnmatch\nimport json\nimport getpass\nimport os\nimport pathlib\nimport datetime\nimport laspy\n\n\nimport time\nimport requests\n                    \nfrom dask.distributed import LocalCluster, SSHCluster \nfrom laserfarm import Retiler, DataProcessing, GeotiffWriter, MacroPipeline\nfrom laserfarm.remote_utils import get_wdclient, get_info_remote, list_remote",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 1,
                "outputs": [],
                "id": "9e76e26f-3a9b-419b-8c96-1a5a188fc79b"
            },
            {
                "cell_type": "markdown",
                "source": "## Global Configuration\u00b6",
                "metadata": {},
                "id": "7e32d454-23ea-4849-a867-8877f7c32dc1"
            },
            {
                "cell_type": "code",
                "source": "# Configurations\n\nimport fnmatch\nimport json\nimport getpass\nimport os\nimport pathlib\nimport datetime\n                    \nfrom dask.distributed import LocalCluster, SSHCluster \nfrom laserfarm import Retiler, DataProcessing, GeotiffWriter, MacroPipeline\nfrom laserfarm.remote_utils import get_wdclient, get_info_remote, list_remote\n\nparam_username = 'USER'\nif 'JUPYTERHUB_USER' in os.environ:\n    param_username = os.environ['JUPYTERHUB_USER']\n    \nconf_remote_path_root = '/webdav/vl-laserfarm'\nconf_remote_path_split = pathlib.Path(conf_remote_path_root + '/split_'+param_username)\nconf_remote_path_retiled = pathlib.Path(conf_remote_path_root + '/retiled_'+param_username)\nconf_remote_path_norm = pathlib.Path(conf_remote_path_root + '/norm_'+param_username)\nconf_remote_path_targets = pathlib.Path(conf_remote_path_root + '/targets_'+param_username)\nconf_remote_path_geotiffs = pathlib.Path(conf_remote_path_root + '/geotiffs_'+param_username)\nconf_local_tmp = pathlib.Path('/tmp')\nconf_remote_path_ahn = pathlib.Path(conf_remote_path_root+'/ahn')\n\n\nparam_hostname = 'https://lfw-ds001-i040.lifewatch.dev/'\nparam_login = 'yifang'\nparam_password = '6xqQziCqdR9a4bjwFuWXBXAvD'\n\nconf_feature_name = 'perc_95_normalized_height'\nconf_validate_precision = '0.001'\nconf_tile_mesh_size = '10'\nconf_filter_type= 'select_equal'\nconf_attribute = 'raw_classification'\nconf_min_x = '-113107.81'\nconf_max_x = '398892.19'\nconf_min_y = '214783.87'\nconf_max_y = '726783.87'\nconf_n_tiles_side = '512'\nconf_apply_filter_value = '1'\nconf_laz_compression_factor = '7'\nconf_max_filesize = '262144000'  # desired max file size (in bytes)\n\nconf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 2,
                "outputs": [],
                "id": "f73fbe04-b8cc-4d5e-8a94-d54d1750d15a"
            },
            {
                "cell_type": "markdown",
                "source": "## Fetching Laz Files from remote WebDAV",
                "metadata": {},
                "id": "5064b6eb-e738-448e-8561-78b0ad7b3734"
            },
            {
                "cell_type": "code",
                "source": "# S1 Fetch Laz Files\n\nlaz_files = [f for f in list_remote(get_wdclient(conf_wd_opts), conf_remote_path_ahn.as_posix())\n             if f.lower().endswith('.laz')]\nprint(laz_files)",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 3,
                "outputs": [
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "['AHN3_T2.laz', 'AHN3_T1.laz']\n"
                    }
                ],
                "id": "f3cf88e5-6d9f-44d1-b5e5-c8e67734f071"
            },
            {
                "cell_type": "markdown",
                "source": "## Splitting big files into smaller ones",
                "metadata": {},
                "id": "1a3cf382-8d3b-4f2a-aa61-f267240709c7"
            },
            {
                "cell_type": "code",
                "source": "# S2 split big files \n\nimport numpy as np\n\ndef save_chunk_to_laz_file(in_filename, \n                           out_filename, \n                           offset, \n                           n_points):\n    \"\"\"Read points from a LAS/LAZ file and write them to a new file.\"\"\"\n    \n    points = np.array([])\n    \n    with laspy.open(in_filename) as in_file:\n        with laspy.open(out_filename, \n                        mode=\"w\", \n                        header=in_file.header) as out_file:\n            in_file.seek(offset)\n            points = in_file.read_points(n_points)\n            out_file.write_points(points)\n    return len(points)\n\ndef split_strategy(filename, max_filesize):\n    \"\"\"Set up splitting strategy for a LAS/LAZ file.\"\"\"\n    with laspy.open(filename) as f:\n        bytes_per_point = (\n            f.header.point_format.num_standard_bytes +\n            f.header.point_format.num_extra_bytes\n        )\n        n_points = f.header.point_count\n    n_points_target = int(\n        max_filesize * int(conf_laz_compression_factor) / bytes_per_point\n    )\n    stem, ext = os.path.splitext(filename)\n    return [\n        (filename, f\"{stem}-{n}{ext}\", offset, n_points_target)\n        for n, offset in enumerate(range(0, n_points, n_points_target))\n    ]\n\nfrom webdav3.client import Client\n\nclient = Client(conf_wd_opts)\nclient.mkdir(conf_remote_path_split.as_posix())\n\n\nremote_path_split = conf_remote_path_split\n\n\nfor file in laz_files:\n    print('Splitting: '+file)\n    client.download_sync(remote_path=os.path.join(conf_remote_path_ahn,file), local_path=file)\n    inps = split_strategy(file, int(conf_max_filesize))\n    for inp in inps:\n        save_chunk_to_laz_file(*inp)\n    client.upload_sync(remote_path=os.path.join(conf_remote_path_split,file), local_path=file)\n\n    for f in os.listdir('.'):\n        if not f.endswith('.LAZ'):\n            continue\n        os.remove(os.path.join('.', f))\n    \n#split_laz_files = laz_files\nremote_path_retiled = str(conf_remote_path_retiled)",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 4,
                "outputs": [
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Splitting: AHN3_T2.laz\nSplitting: AHN3_T1.laz\n"
                    }
                ],
                "id": "599581f0-fb85-4d2d-a658-789a4919ef07"
            },
            {
                "cell_type": "markdown",
                "source": "## Retiling",
                "metadata": {},
                "id": "0ac19b05-e6be-42ae-8239-0cc7ffbece8f"
            },
            {
                "cell_type": "code",
                "source": "# S3 Read splitted Laz Files\nremote_path_retiled\n\nsplit_laz_files = [f for f in list_remote(get_wdclient(conf_wd_opts), pathlib.Path(conf_remote_path_split).as_posix())\n             if f.lower().endswith('.laz')]",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 5,
                "outputs": [],
                "id": "38afc563-d4a5-45ba-9546-8aea1c2bd7fa"
            },
            {
                "cell_type": "code",
                "source": "# S4 Retiling\n\nfrom IPython.display import display, HTML   \n\nremote_path_split = conf_remote_path_split\n\ngrid_retile = {\n    'min_x': float(conf_min_x),\n    'max_x': float(conf_max_x),\n    'min_y': float(conf_min_y),\n    'max_y': float(conf_max_y),\n    'n_tiles_side': int(conf_n_tiles_side)\n}\n\n\nretiling_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_split.as_posix(),\n    'set_grid': grid_retile,\n    'split_and_redistribute': {},\n    'validate': {},\n    'pushremote': conf_remote_path_retiled.as_posix(),\n    'cleanlocalfs': {}\n}\n\n#file = split_laz_files[0]    \n#retiler = Retiler(file,label=file).config(retiling_input).setup_webdav_client(conf_wd_opts)\n\nfor file in split_laz_files:\n    print('Retiling: '+file)\n    retiler = Retiler(file, label=file).config(retiling_input).setup_webdav_client(conf_wd_opts)\n    retiler_output = retiler.run()\n    \nremote_path_norm = str(conf_remote_path_norm)",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 6,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:07,249 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/AHN3_T2.laz_input\n2023-11-14 14:07:07,251 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/AHN3_T2.laz_output\n2023-11-14 14:07:07,259 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/split_USER/AHN3_T2.laz ...\n"
                    },
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Retiling: AHN3_T2.laz\n"
                    },
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:11,819 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:07:11,821 -                        laserfarm.retiler -       INFO - Setting up the target grid\n2023-11-14 14:07:11,823 -                        laserfarm.retiler -       INFO - Splitting file /tmp/AHN3_T2.laz_input/AHN3_T2.laz with PDAL ...\n2023-11-14 14:07:35,928 -                        laserfarm.retiler -       INFO - ... splitting completed.\n2023-11-14 14:07:35,934 -                        laserfarm.retiler -       INFO - Redistributing files to tiles ...\n2023-11-14 14:07:35,937 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_4.laz to tile_289_272\n2023-11-14 14:07:35,939 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_3.laz to tile_288_272\n2023-11-14 14:07:35,942 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_1.laz to tile_289_271\n2023-11-14 14:07:35,944 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_2.laz to tile_288_271\n2023-11-14 14:07:35,945 -                        laserfarm.retiler -       INFO - ... redistributing completed.\n2023-11-14 14:07:35,947 -                        laserfarm.retiler -       INFO - Validating split ...\n2023-11-14 14:07:35,948 -                        laserfarm.retiler -       INFO - ... 13973048 points in parent file\n2023-11-14 14:07:35,950 -                        laserfarm.retiler -       INFO - ... 528208 points in AHN3_T2_4.laz\n2023-11-14 14:07:35,951 -                        laserfarm.retiler -       INFO - ... 9582849 points in AHN3_T2_2.laz\n2023-11-14 14:07:35,952 -                        laserfarm.retiler -       INFO - ... 1450445 points in AHN3_T2_1.laz\n2023-11-14 14:07:35,957 -                        laserfarm.retiler -       INFO - ... 2411546 points in AHN3_T2_3.laz\n2023-11-14 14:07:35,958 -                        laserfarm.retiler -       INFO - ... split validation completed.\n2023-11-14 14:07:35,960 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/retiled_USER ...\n2023-11-14 14:07:53,426 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:07:53,428 -           laserfarm.pipeline_remote_data -       INFO - Removing input and output folders\n2023-11-14 14:07:53,456 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/AHN3_T1.laz_input\n2023-11-14 14:07:53,458 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/AHN3_T1.laz_output\n2023-11-14 14:07:53,460 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/split_USER/AHN3_T1.laz ...\n"
                    },
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Retiling: AHN3_T1.laz\n"
                    },
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:58,926 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:07:58,928 -                        laserfarm.retiler -       INFO - Setting up the target grid\n2023-11-14 14:07:58,930 -                        laserfarm.retiler -       INFO - Splitting file /tmp/AHN3_T1.laz_input/AHN3_T1.laz with PDAL ...\n2023-11-14 14:08:44,639 -                        laserfarm.retiler -       INFO - ... splitting completed.\n2023-11-14 14:08:44,642 -                        laserfarm.retiler -       INFO - Redistributing files to tiles ...\n2023-11-14 14:08:44,645 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_3.laz to tile_289_271\n2023-11-14 14:08:44,647 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_2.laz to tile_289_270\n2023-11-14 14:08:44,649 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_4.laz to tile_288_271\n2023-11-14 14:08:44,651 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_1.laz to tile_288_270\n2023-11-14 14:08:44,652 -                        laserfarm.retiler -       INFO - ... redistributing completed.\n2023-11-14 14:08:44,653 -                        laserfarm.retiler -       INFO - Validating split ...\n2023-11-14 14:08:44,654 -                        laserfarm.retiler -       INFO - ... 17822698 points in parent file\n2023-11-14 14:08:44,655 -                        laserfarm.retiler -       INFO - ... 2563278 points in AHN3_T1_4.laz\n2023-11-14 14:08:44,656 -                        laserfarm.retiler -       INFO - ... 603139 points in AHN3_T1_3.laz\n2023-11-14 14:08:44,658 -                        laserfarm.retiler -       INFO - ... 12765152 points in AHN3_T1_1.laz\n2023-11-14 14:08:44,659 -                        laserfarm.retiler -       INFO - ... 1891129 points in AHN3_T1_2.laz\n2023-11-14 14:08:44,660 -                        laserfarm.retiler -       INFO - ... split validation completed.\n2023-11-14 14:08:44,661 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/retiled_USER ...\n2023-11-14 14:09:02,425 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:09:02,427 -           laserfarm.pipeline_remote_data -       INFO - Removing input and output folders\n"
                    }
                ],
                "id": "d33b3708-c431-40a5-9e69-10b3c7777e85"
            },
            {
                "cell_type": "markdown",
                "source": "## Normalization",
                "metadata": {},
                "id": "cf7fdf79-530d-41a7-88c3-1177d1a088b2"
            },
            {
                "cell_type": "code",
                "source": "# S5 Fetch Tiles for Norm\nremote_path_norm\n\ntiles = [t.strip('/') for t in list_remote(get_wdclient(conf_wd_opts), conf_remote_path_retiled.as_posix())\n         if fnmatch.fnmatch(t, 'tile_*_*/')]",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 7,
                "outputs": [],
                "id": "8e8cfd3a-288d-43f2-9f69-298ec0b28751"
            },
            {
                "cell_type": "code",
                "source": "# S6 Normalization\n\nimport copy\n\ntiles\n\nremote_path_norm = str(conf_remote_path_norm)\n\nnormalization_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_retiled.as_posix(),\n    'load': {'attributes': 'all'},\n    # Filter out artifically high points - give overflow error when writing\n    'apply_filter': {'filter_type':'select_below',\n                     'attribute': 'z',\n                     'threshold': 10000.},  # remove non-physically heigh points\n    'normalize': 1,\n    'clear_cache' : {},\n    'pushremote': conf_remote_path_norm.as_posix(),\n}\n\n# write input dictionary to JSON file\nwith open('normalize.json', 'w') as f:\n    json.dump(normalization_input, f)\n    \n\n# add pipeline list to macro-pipeline object and set the corresponding labels\n#tile = tiles\n# for tile in tiles:\n#normalization_input_ = copy.deepcopy(normalization_input)\n#normalization_input_['export_point_cloud'] = {'filename': '{}.laz'.format(tile),'overwrite': True}\n#dp = DataProcessing(tile, label=tile).config(normalization_input_).setup_webdav_client(conf_wd_opts)\n\nfor tile in tiles:\n    normalization_input_ = copy.deepcopy(normalization_input)\n    normalization_input_['export_point_cloud'] = {'filename': '{}.laz'.format(tile),'overwrite': True}\n    dp = DataProcessing(tile, label=tile).config(normalization_input_).setup_webdav_client(conf_wd_opts)\n    dp.run()\n\nremote_path_norm\n",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 8,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:09:03,043 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_271_input\n2023-11-14 14:09:03,045 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_271_output\n2023-11-14 14:09:03,049 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_271 ...\n2023-11-14 14:09:07,069 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:09:07,071 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:09:07,072 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_271_input/tile_289_271/AHN3_T1_3.laz\n2023-11-14 14:09:07,478 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_271_input/tile_289_271/AHN3_T2_1.laz\n2023-11-14 14:09:20,918 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:09:20,919 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:09:21,164 -                                     root -       INFO - Cylinder size in Bytes: 692155693.4389035\n2023-11-14 14:09:21,165 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:09:21,167 -                                     root -       INFO - Start tree creation\n2023-11-14 14:09:25,841 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:09:25,855 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:09:34,368 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:09:34,370 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:09:46,418 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:09:46,421 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_271_output/tile_289_271.laz\n2023-11-14 14:10:11,878 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:10:11,883 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:10:11,886 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:10:17,603 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:10:17,610 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_271_input\n2023-11-14 14:10:17,612 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_271_output\n2023-11-14 14:10:17,613 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_271 ...\n2023-11-14 14:10:23,807 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:10:23,810 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:10:23,811 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_271_input/tile_288_271/AHN3_T1_4.laz\n2023-11-14 14:10:25,596 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_271_input/tile_288_271/AHN3_T2_2.laz\n2023-11-14 14:11:06,236 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:11:06,238 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:11:07,911 -                                     root -       INFO - Cylinder size in Bytes: 5723102168.897599\n2023-11-14 14:11:07,913 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:11:07,914 -                                     root -       INFO - Start tree creation\n2023-11-14 14:11:10,516 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:11:10,623 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:12:09,087 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:12:09,088 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:12:10,448 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:12:10,449 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_271_output/tile_288_271.laz\n2023-11-14 14:12:21,933 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:12:21,934 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:12:21,957 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:12:33,642 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:12:33,663 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_272_input\n2023-11-14 14:12:33,665 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_272_output\n2023-11-14 14:12:33,668 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_272 ...\n2023-11-14 14:12:36,571 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:12:36,572 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:12:36,573 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_272_input/tile_288_272/AHN3_T2_3.laz\n2023-11-14 14:12:38,182 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:12:38,184 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:12:38,620 -                                     root -       INFO - Cylinder size in Bytes: 1241913170.650779\n2023-11-14 14:12:38,622 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:12:38,624 -                                     root -       INFO - Start tree creation\n2023-11-14 14:12:39,033 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:12:39,061 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:12:50,971 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:12:50,974 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:12:51,062 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:12:51,064 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_272_output/tile_288_272.laz\n2023-11-14 14:12:53,077 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:12:53,078 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:12:53,081 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:12:57,999 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:12:58,010 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_272_input\n2023-11-14 14:12:58,011 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_272_output\n2023-11-14 14:12:58,013 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_272 ...\n2023-11-14 14:13:00,562 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:13:00,564 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:13:00,565 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_272_input/tile_289_272/AHN3_T2_4.laz\n2023-11-14 14:13:00,921 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:13:00,922 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:13:00,973 -                                     root -       INFO - Cylinder size in Bytes: 150197785.47624204\n2023-11-14 14:13:00,974 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:13:00,975 -                                     root -       INFO - Start tree creation\n2023-11-14 14:13:01,066 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:13:01,070 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:13:03,304 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:13:03,306 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:13:03,317 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:13:03,318 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_272_output/tile_289_272.laz\n2023-11-14 14:13:03,726 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:13:03,728 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:13:03,728 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:13:07,560 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:13:07,568 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_270_input\n2023-11-14 14:13:07,569 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_270_output\n2023-11-14 14:13:07,572 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_270 ...\n2023-11-14 14:13:12,688 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:13:12,691 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:13:12,692 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_270_input/tile_288_270/AHN3_T1_1.laz\n2023-11-14 14:13:22,049 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:13:22,050 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:13:23,300 -                                     root -       INFO - Cylinder size in Bytes: 4486912100.415718\n2023-11-14 14:13:23,301 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:13:23,302 -                                     root -       INFO - Start tree creation\n2023-11-14 14:13:25,700 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:13:25,795 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:14:21,239 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:14:21,243 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:14:22,442 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:14:22,448 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_270_output/tile_288_270.laz\n2023-11-14 14:14:34,067 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:14:34,068 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:14:34,084 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:14:46,022 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:14:46,057 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_270_input\n2023-11-14 14:14:46,059 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_270_output\n2023-11-14 14:14:46,061 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_270 ...\n2023-11-14 14:14:48,934 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:14:48,937 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:14:48,938 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_270_input/tile_289_270/AHN3_T1_2.laz\n2023-11-14 14:14:50,295 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:14:50,297 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:14:50,452 -                                     root -       INFO - Cylinder size in Bytes: 542650063.6561003\n2023-11-14 14:14:50,453 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:14:50,454 -                                     root -       INFO - Start tree creation\n2023-11-14 14:14:50,790 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:14:50,803 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:14:58,704 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:14:58,708 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:14:58,798 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:14:58,799 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_270_output/tile_289_270.laz\n2023-11-14 14:15:00,478 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:15:00,480 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:15:00,483 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:15:05,681 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n"
                    },
                    {
                        "execution_count": 8,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "'/webdav/vl-laserfarm/norm_USER'"
                        },
                        "metadata": {}
                    }
                ],
                "id": "365ab2e8-c194-4828-90cd-7d91fe815b58"
            },
            {
                "cell_type": "markdown",
                "source": "## Feature extraction",
                "metadata": {},
                "id": "caa87bfe-cfe8-4aea-a292-e4d11e479dd4"
            },
            {
                "cell_type": "code",
                "source": "# S7 Fetch Norm Tiles for feature\nremote_path_norm\n\ntiles = [f for f in list_remote(get_wdclient(conf_wd_opts), pathlib.Path(conf_remote_path_norm).as_posix())\n             if f.lower().endswith('.laz')]\n\ntiles",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 9,
                "outputs": [
                    {
                        "execution_count": 9,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "['tile_289_271.laz',\n 'tile_288_271.laz',\n 'tile_288_272.laz',\n 'tile_289_272.laz',\n 'tile_288_270.laz',\n 'tile_289_270.laz']"
                        },
                        "metadata": {}
                    }
                ],
                "id": "71329832-64e6-406d-9f03-e1f013754308"
            },
            {
                "cell_type": "code",
                "source": "# S8 Feature Extraction\n\nremote_path_targets = str(conf_remote_path_targets)\n\nfeatures = [\"perc_95_normalized_height\"]\n\ntile_mesh_size = float(conf_tile_mesh_size)\n\ngrid_feature = {\n    'min_x': float(conf_min_x),\n    'max_x': float(conf_max_x),\n    'min_y': float(conf_min_y),\n    'max_y': float(conf_max_y),\n    'n_tiles_side': int(conf_n_tiles_side)\n}\n\nfeature_extraction_input = {\n    'setup_local_fs': {\n        'input_folder': (conf_local_tmp / 'tile_input').as_posix(),\n        'output_folder': (conf_local_tmp / 'tile_output').as_posix(),\n    },\n    'pullremote': conf_remote_path_norm.as_posix(),\n    'load': {'attributes': [conf_attribute]},\n    'normalize': 1,\n    'apply_filter': {\n        'filter_type': conf_filter_type, \n        'attribute': conf_attribute,\n        'value': [int(conf_apply_filter_value)]#ground surface (2), water (9), buildings (6), artificial objects (26), vegetation (?), and unclassified (1)\n    },\n    'generate_targets': {\n        'tile_mesh_size' : tile_mesh_size,\n        'validate' : True,\n        'validate_precision': float(conf_validate_precision),\n        **grid_feature\n    },\n    'extract_features': {\n        'feature_names': features,\n        'volume_type': 'cell',\n        'volume_size': tile_mesh_size\n    },\n    'export_targets': {\n        'attributes': features,\n        'multi_band_files': False\n    },\n    'pushremote': conf_remote_path_targets.as_posix(),\n#     'cleanlocalfs': {}\n}\n\nfor t in tiles:    \n    idx = (t.split('.')[0].split('_')[1:])\n    processing = DataProcessing(t, tile_index=idx,label=t).config(feature_extraction_input).setup_webdav_client(conf_wd_opts)\n    processing.run()\n\nremote_path_targets",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 10,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:15:06,342 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:15:06,343 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:15:06,344 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_271.laz ...\n2023-11-14 14:15:08,991 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:15:08,993 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:15:08,994 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_271.laz\n2023-11-14 14:15:11,073 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:15:11,074 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:15:11,285 -                                     root -       INFO - Cylinder size in Bytes: 692155693.4389035\n2023-11-14 14:15:11,287 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:11,292 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:11,674 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:11,691 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:15:20,371 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:15:20,373 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:15:20,431 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:15:20,433 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,271)\n2023-11-14 14:15:20,452 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:15:20,455 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:15:20,456 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:15:20,457 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:15:20,460 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:15:20,461 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:20,462 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:20,607 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:20,610 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:15:22,663 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:15:22,808 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.14 seconds\n2023-11-14 14:15:22,816 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:15:22,839 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:15:22,841 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_271.ply\n2023-11-14 14:15:22,959 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:15:22,960 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:15:27,518 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:15:27,527 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:15:27,528 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:15:27,530 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_271.laz ...\n2023-11-14 14:15:34,971 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:15:34,973 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:15:34,973 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_271.laz\n2023-11-14 14:15:47,228 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:15:47,230 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:15:48,887 -                                     root -       INFO - Cylinder size in Bytes: 5723102168.897599\n2023-11-14 14:15:48,889 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:48,891 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:51,042 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:51,155 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:16:49,085 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:16:49,088 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:16:49,337 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:16:49,339 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,271)\n2023-11-14 14:16:49,417 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:16:49,420 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:16:49,421 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:16:49,422 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:16:49,425 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:16:49,426 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:16:49,427 -                                     root -       INFO - Start tree creation\n2023-11-14 14:16:49,962 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:16:49,966 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:16:56,213 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:16:56,681 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.47 seconds\n2023-11-14 14:16:56,703 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:16:56,755 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:16:56,757 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_271.ply\n2023-11-14 14:16:56,888 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:16:56,889 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:03,871 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:03,886 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:03,888 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:03,891 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_272.laz ...\n2023-11-14 14:17:06,555 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:06,559 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:06,560 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_272.laz\n2023-11-14 14:17:08,896 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:17:08,898 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:17:09,252 -                                     root -       INFO - Cylinder size in Bytes: 1241913170.650779\n2023-11-14 14:17:09,254 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:09,255 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:09,660 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:09,686 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:21,395 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:17:21,397 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:17:21,424 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:17:21,425 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,272)\n2023-11-14 14:17:21,431 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:17:21,433 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:17:21,434 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:17:21,436 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:17:21,437 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:17:21,438 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:21,439 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:21,477 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:21,480 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:21,977 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:17:22,041 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.06 seconds\n2023-11-14 14:17:22,044 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:17:22,049 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:17:22,051 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_272.ply\n2023-11-14 14:17:22,170 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:17:22,171 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:31,860 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:31,868 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:31,869 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:31,872 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_272.laz ...\n2023-11-14 14:17:33,660 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:33,662 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:33,663 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_272.laz\n2023-11-14 14:17:34,209 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:17:34,211 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:17:34,255 -                                     root -       INFO - Cylinder size in Bytes: 150197785.47624204\n2023-11-14 14:17:34,256 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:34,257 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:34,344 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:34,348 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:36,287 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:17:36,289 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:17:36,310 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:17:36,312 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,272)\n2023-11-14 14:17:36,316 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:17:36,317 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:17:36,318 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:17:36,319 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:17:36,320 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:17:36,321 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:36,322 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:36,380 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:36,384 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:37,099 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:17:37,160 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.06 seconds\n2023-11-14 14:17:37,163 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:17:37,171 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:17:37,172 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_272.ply\n2023-11-14 14:17:37,297 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:17:37,298 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:49,909 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:49,916 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:49,917 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:49,921 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_270.laz ...\n2023-11-14 14:17:58,019 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:58,021 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:58,022 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_270.laz\n2023-11-14 14:18:12,472 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:18:12,478 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:18:13,831 -                                     root -       INFO - Cylinder size in Bytes: 4486912100.415718\n2023-11-14 14:18:13,833 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:18:13,834 -                                     root -       INFO - Start tree creation\n2023-11-14 14:18:16,697 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:18:16,783 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:11,409 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:19:11,413 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:19:12,098 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:19:12,101 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,270)\n2023-11-14 14:19:12,213 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:19:12,215 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:19:12,216 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:19:12,217 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:19:12,220 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:19:12,221 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:12,222 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:12,929 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:12,933 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:21,551 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:19:22,165 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.61 seconds\n2023-11-14 14:19:22,195 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:19:22,271 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:19:22,274 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_270.ply\n2023-11-14 14:19:22,419 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:19:22,420 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:19:39,549 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:19:39,566 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:19:39,567 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:19:39,576 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_270.laz ...\n2023-11-14 14:19:42,000 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:19:42,001 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:19:42,004 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_270.laz\n2023-11-14 14:19:43,821 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:19:43,852 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:19:44,008 -                                     root -       INFO - Cylinder size in Bytes: 542650063.6561003\n2023-11-14 14:19:44,009 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:44,011 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:44,333 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:44,346 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:51,518 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:19:51,520 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:19:51,553 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:19:51,555 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,270)\n2023-11-14 14:19:51,564 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:19:51,566 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:19:51,568 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:19:51,569 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:19:51,571 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:19:51,572 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:51,572 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:51,646 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:51,650 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:52,853 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:19:52,968 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.11 seconds\n2023-11-14 14:19:52,974 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:19:52,985 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:19:52,987 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_270.ply\n2023-11-14 14:19:53,120 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:19:53,121 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:20:13,366 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n"
                    },
                    {
                        "execution_count": 10,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "'/webdav/vl-laserfarm/targets_USER'"
                        },
                        "metadata": {}
                    }
                ],
                "id": "91bbaa1c-9d83-4703-9477-dcd88522ded2"
            },
            {
                "cell_type": "markdown",
                "source": "## GeoTIFF export",
                "metadata": {
                    "tags": []
                },
                "id": "2eadeb04-efe1-4b68-810b-efd4a91b8c18"
            },
            {
                "cell_type": "code",
                "source": "# S9 GeoTIFF Export\n\nremote_path_targets\n\nremote_path_geotiffs = conf_remote_path_geotiffs\n\n# setup input dictionary to configure the GeoTIFF export pipeline\ngeotiff_export_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_targets.as_posix(),\n    'parse_point_cloud': {},\n    'data_split': {'xSub': 1, 'ySub': 1},\n    'create_subregion_geotiffs': {'output_handle': 'geotiff'},\n    'pushremote': remote_path_geotiffs.as_posix(),\n    'cleanlocalfs': {}   \n}\n\nwriter = GeotiffWriter(input_dir=conf_feature_name, bands=conf_feature_name,label=conf_feature_name).config(geotiff_export_input).setup_webdav_client(conf_wd_opts)\nwriter.run()",
                "metadata": {
                    "trusted": true
                },
                "execution_count": null,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:20:13,387 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/perc_95_normalized_height_input\n2023-11-14 14:20:13,388 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/perc_95_normalized_height_output\n2023-11-14 14:20:13,391 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/targets_USER/perc_95_normalized_height ...\n"
                    }
                ],
                "id": "cdd5ac0b-78e7-40c8-81ff-0078d3916ecd"
            },
            {
                "cell_type": "code",
                "source": "",
                "metadata": {
                    "trusted": true
                },
                "execution_count": null,
                "outputs": [],
                "id": "08ea3b0b-8627-44ab-a8da-56d7ce531e7e"
            }
        ]
    }
[I 2023-11-16 10:06:33.264 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/extract?1700129193093 (anonymous@10.244.7.0) 48.89ms
[I 2023-11-16 10:06:34.368 ServerApp] 200 GET /vl-laserfarm/user/USER/api/terminals?1700129194340 (USER@10.244.9.0) 2.96ms
[I 2023-11-16 10:06:34.625 ServerApp] 200 GET /vl-laserfarm/user/USER/api/contents?content=1&1700129194559 (USER@10.244.11.0) 34.94ms
[I 2023-11-16 10:06:35.960 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129195929 (anonymous@10.244.4.1) 2.02ms
[I 2023-11-16 10:06:36.450 ServerApp] 200 GET /vl-laserfarm/user/USER/api/sessions?1700129196415 (USER@10.244.0.0) 2.71ms
[I 2023-11-16 10:06:36.512 ServerApp] 200 GET /vl-laserfarm/user/USER/api/kernels?1700129196433 (USER@10.244.7.0) 2.78ms
[I 2023-11-16 10:06:37.467 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129197439 (anonymous@10.244.9.0) 2.48ms
[I 2023-11-16 10:06:39.194 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129199164 (anonymous@10.244.11.0) 2.22ms
[I 2023-11-16 10:06:40.457 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129200423 (anonymous@10.244.4.1) 2.15ms
[I 2023-11-16 10:06:42.283 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129202244 (anonymous@10.244.0.0) 2.26ms
[I 2023-11-16 10:06:44.437 ServerApp] 200 GET /vl-laserfarm/user/USER/api/terminals?1700129204381 (USER@10.244.7.0) 1.56ms
[I 2023-11-16 10:06:44.565 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129204535 (anonymous@10.244.9.0) 1.23ms
[I 2023-11-16 10:06:44.684 ServerApp] 200 GET /vl-laserfarm/user/USER/api/contents?content=1&1700129204638 (USER@10.244.11.0) 24.07ms
[I 2023-11-16 10:06:46.511 ServerApp] 200 GET /vl-laserfarm/user/USER/api/sessions?1700129206475 (USER@10.244.4.1) 3.32ms
[I 2023-11-16 10:06:46.599 ServerApp] 200 GET /vl-laserfarm/user/USER/api/kernels?1700129206557 (USER@10.244.0.0) 1.56ms
[I 2023-11-16 10:06:46.730 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/baseimage?1700129206695 (anonymous@10.244.7.0) 2.07ms
/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.github.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
[I 2023-11-16 10:06:51.239 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/addcell?1700129207467 (anonymous@10.244.9.0) 3749.29ms
}
--------------------------------------
current_cell: {
    "all_inputs": [
        "remote_path_norm"
    ],
    "base_image": "qcdis/miniconda3-pdal",
    "chart_obj": {
        "hovered": {},
        "links": {},
        "nodes": {
            "e577538": {
                "id": "e577538",
                "ports": {
                    "remote_path_norm": {
                        "id": "remote_path_norm",
                        "properties": {
                            "color": "#a1c587"
                        },
                        "type": "left"
                    },
                    "tiles": {
                        "id": "tiles",
                        "properties": {
                            "color": "#406abf"
                        },
                        "type": "right"
                    }
                },
                "position": {
                    "x": 35,
                    "y": 15
                },
                "properties": {
                    "deps": [
                        "",
                        "laserfarm"
                    ],
                    "inputs": [
                        "remote_path_norm"
                    ],
                    "og_node_id": "e577538",
                    "outputs": [
                        "tiles"
                    ],
                    "params": [
                        "param_password",
                        "param_login",
                        "param_username",
                        "param_hostname"
                    ],
                    "title": "S7 Fetch Norm Tiles for feature-USER",
                    "vars": [
                        {
                            "color": "#a1c587",
                            "direction": "input",
                            "name": "remote_path_norm",
                            "type": "datatype"
                        },
                        {
                            "color": "#406abf",
                            "direction": "output",
                            "name": "tiles",
                            "type": "datatype"
                        }
                    ]
                },
                "type": "input-output"
            }
        },
        "offset": {
            "x": 0,
            "y": 0
        },
        "scale": 1,
        "selected": {}
    },
    "confs": {
        "conf_remote_path_norm": "conf_remote_path_norm = pathlib.Path( '/webdav/vl-laserfarm' + '/norm_'+param_username)",
        "conf_wd_opts": "conf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}"
    },
    "container_source": "",
    "dependencies": [
        {
            "asname": null,
            "module": "",
            "name": "pathlib"
        },
        {
            "asname": null,
            "module": "laserfarm.remote_utils",
            "name": "list_remote"
        },
        {
            "asname": null,
            "module": "laserfarm.remote_utils",
            "name": "get_wdclient"
        }
    ],
    "inputs": [
        "remote_path_norm"
    ],
    "kernel": "ipython",
    "node_id": "e577538",
    "original_source": "conf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}\nconf_remote_path_norm = pathlib.Path( '/webdav/vl-laserfarm' + '/norm_'+param_username)\nremote_path_norm\n\ntiles = [f for f in list_remote(get_wdclient(conf_wd_opts), pathlib.Path(conf_remote_path_norm).as_posix())\n             if f.lower().endswith('.laz')]\n\ntiles",
    "outputs": [
        "tiles"
    ],
    "params": [
        "param_password",
        "param_login",
        "param_username",
        "param_hostname"
    ],
    "task_name": "s7-fetch-norm-tiles-for-feature-USER",
    "title": "S7 Fetch Norm Tiles for feature-USER",
    "types": {
        "param_hostname": "str",
        "param_login": "str",
        "param_password": "str",
        "param_username": "str",
        "remote_path_norm": "str",
        "tiles": "list"
    }
}
--------------------------------------
ExtractorHandler. payload: {
    "save": false,
    "kernel": "ipython",
    "cell_index": 15,
    "notebook": {
        "metadata": {
            "kernelspec": {
                "display_name": "Python 3 (ipykernel)",
                "language": "python",
                "name": "python3"
            },
            "language_info": {
                "codemirror_mode": {
                    "name": "ipython",
                    "version": 3
                },
                "file_extension": ".py",
                "mimetype": "text/x-python",
                "name": "python",
                "nbconvert_exporter": "python",
                "pygments_lexer": "ipython3",
                "version": "3.9.16"
            }
        },
        "nbformat_minor": 5,
        "nbformat": 4,
        "cells": [
            {
                "cell_type": "code",
                "source": "import fnmatch\nimport json\nimport getpass\nimport os\nimport pathlib\nimport datetime\nimport laspy\n\n\nimport time\nimport requests\n                    \nfrom dask.distributed import LocalCluster, SSHCluster \nfrom laserfarm import Retiler, DataProcessing, GeotiffWriter, MacroPipeline\nfrom laserfarm.remote_utils import get_wdclient, get_info_remote, list_remote",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 1,
                "outputs": [],
                "id": "9e76e26f-3a9b-419b-8c96-1a5a188fc79b"
            },
            {
                "cell_type": "markdown",
                "source": "## Global Configuration\u00b6",
                "metadata": {},
                "id": "7e32d454-23ea-4849-a867-8877f7c32dc1"
            },
            {
                "cell_type": "code",
                "source": "# Configurations\n\nimport fnmatch\nimport json\nimport getpass\nimport os\nimport pathlib\nimport datetime\n                    \nfrom dask.distributed import LocalCluster, SSHCluster \nfrom laserfarm import Retiler, DataProcessing, GeotiffWriter, MacroPipeline\nfrom laserfarm.remote_utils import get_wdclient, get_info_remote, list_remote\n\nparam_username = 'USER'\nif 'JUPYTERHUB_USER' in os.environ:\n    param_username = os.environ['JUPYTERHUB_USER']\n    \nconf_remote_path_root = '/webdav/vl-laserfarm'\nconf_remote_path_split = pathlib.Path(conf_remote_path_root + '/split_'+param_username)\nconf_remote_path_retiled = pathlib.Path(conf_remote_path_root + '/retiled_'+param_username)\nconf_remote_path_norm = pathlib.Path(conf_remote_path_root + '/norm_'+param_username)\nconf_remote_path_targets = pathlib.Path(conf_remote_path_root + '/targets_'+param_username)\nconf_remote_path_geotiffs = pathlib.Path(conf_remote_path_root + '/geotiffs_'+param_username)\nconf_local_tmp = pathlib.Path('/tmp')\nconf_remote_path_ahn = pathlib.Path(conf_remote_path_root+'/ahn')\n\n\nparam_hostname = 'https://lfw-ds001-i040.lifewatch.dev/'\nparam_login = 'yifang'\nparam_password = '6xqQziCqdR9a4bjwFuWXBXAvD'\n\nconf_feature_name = 'perc_95_normalized_height'\nconf_validate_precision = '0.001'\nconf_tile_mesh_size = '10'\nconf_filter_type= 'select_equal'\nconf_attribute = 'raw_classification'\nconf_min_x = '-113107.81'\nconf_max_x = '398892.19'\nconf_min_y = '214783.87'\nconf_max_y = '726783.87'\nconf_n_tiles_side = '512'\nconf_apply_filter_value = '1'\nconf_laz_compression_factor = '7'\nconf_max_filesize = '262144000'  # desired max file size (in bytes)\n\nconf_wd_opts = { 'webdav_hostname': param_hostname, 'webdav_login': param_login, 'webdav_password': param_password}",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 2,
                "outputs": [],
                "id": "f73fbe04-b8cc-4d5e-8a94-d54d1750d15a"
            },
            {
                "cell_type": "markdown",
                "source": "## Fetching Laz Files from remote WebDAV",
                "metadata": {},
                "id": "5064b6eb-e738-448e-8561-78b0ad7b3734"
            },
            {
                "cell_type": "code",
                "source": "# S1 Fetch Laz Files\n\nlaz_files = [f for f in list_remote(get_wdclient(conf_wd_opts), conf_remote_path_ahn.as_posix())\n             if f.lower().endswith('.laz')]\nprint(laz_files)",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 3,
                "outputs": [
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "['AHN3_T2.laz', 'AHN3_T1.laz']\n"
                    }
                ],
                "id": "f3cf88e5-6d9f-44d1-b5e5-c8e67734f071"
            },
            {
                "cell_type": "markdown",
                "source": "## Splitting big files into smaller ones",
                "metadata": {},
                "id": "1a3cf382-8d3b-4f2a-aa61-f267240709c7"
            },
            {
                "cell_type": "code",
                "source": "# S2 split big files \n\nimport numpy as np\n\ndef save_chunk_to_laz_file(in_filename, \n                           out_filename, \n                           offset, \n                           n_points):\n    \"\"\"Read points from a LAS/LAZ file and write them to a new file.\"\"\"\n    \n    points = np.array([])\n    \n    with laspy.open(in_filename) as in_file:\n        with laspy.open(out_filename, \n                        mode=\"w\", \n                        header=in_file.header) as out_file:\n            in_file.seek(offset)\n            points = in_file.read_points(n_points)\n            out_file.write_points(points)\n    return len(points)\n\ndef split_strategy(filename, max_filesize):\n    \"\"\"Set up splitting strategy for a LAS/LAZ file.\"\"\"\n    with laspy.open(filename) as f:\n        bytes_per_point = (\n            f.header.point_format.num_standard_bytes +\n            f.header.point_format.num_extra_bytes\n        )\n        n_points = f.header.point_count\n    n_points_target = int(\n        max_filesize * int(conf_laz_compression_factor) / bytes_per_point\n    )\n    stem, ext = os.path.splitext(filename)\n    return [\n        (filename, f\"{stem}-{n}{ext}\", offset, n_points_target)\n        for n, offset in enumerate(range(0, n_points, n_points_target))\n    ]\n\nfrom webdav3.client import Client\n\nclient = Client(conf_wd_opts)\nclient.mkdir(conf_remote_path_split.as_posix())\n\n\nremote_path_split = conf_remote_path_split\n\n\nfor file in laz_files:\n    print('Splitting: '+file)\n    client.download_sync(remote_path=os.path.join(conf_remote_path_ahn,file), local_path=file)\n    inps = split_strategy(file, int(conf_max_filesize))\n    for inp in inps:\n        save_chunk_to_laz_file(*inp)\n    client.upload_sync(remote_path=os.path.join(conf_remote_path_split,file), local_path=file)\n\n    for f in os.listdir('.'):\n        if not f.endswith('.LAZ'):\n            continue\n        os.remove(os.path.join('.', f))\n    \n#split_laz_files = laz_files\nremote_path_retiled = str(conf_remote_path_retiled)",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 4,
                "outputs": [
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Splitting: AHN3_T2.laz\nSplitting: AHN3_T1.laz\n"
                    }
                ],
                "id": "599581f0-fb85-4d2d-a658-789a4919ef07"
            },
            {
                "cell_type": "markdown",
                "source": "## Retiling",
                "metadata": {},
                "id": "0ac19b05-e6be-42ae-8239-0cc7ffbece8f"
            },
            {
                "cell_type": "code",
                "source": "# S3 Read splitted Laz Files\nremote_path_retiled\n\nsplit_laz_files = [f for f in list_remote(get_wdclient(conf_wd_opts), pathlib.Path(conf_remote_path_split).as_posix())\n             if f.lower().endswith('.laz')]",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 5,
                "outputs": [],
                "id": "38afc563-d4a5-45ba-9546-8aea1c2bd7fa"
            },
            {
                "cell_type": "code",
                "source": "# S4 Retiling\n\nfrom IPython.display import display, HTML   \n\nremote_path_split = conf_remote_path_split\n\ngrid_retile = {\n    'min_x': float(conf_min_x),\n    'max_x': float(conf_max_x),\n    'min_y': float(conf_min_y),\n    'max_y': float(conf_max_y),\n    'n_tiles_side': int(conf_n_tiles_side)\n}\n\n\nretiling_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_split.as_posix(),\n    'set_grid': grid_retile,\n    'split_and_redistribute': {},\n    'validate': {},\n    'pushremote': conf_remote_path_retiled.as_posix(),\n    'cleanlocalfs': {}\n}\n\n#file = split_laz_files[0]    \n#retiler = Retiler(file,label=file).config(retiling_input).setup_webdav_client(conf_wd_opts)\n\nfor file in split_laz_files:\n    print('Retiling: '+file)\n    retiler = Retiler(file, label=file).config(retiling_input).setup_webdav_client(conf_wd_opts)\n    retiler_output = retiler.run()\n    \nremote_path_norm = str(conf_remote_path_norm)",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 6,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:07,249 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/AHN3_T2.laz_input\n2023-11-14 14:07:07,251 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/AHN3_T2.laz_output\n2023-11-14 14:07:07,259 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/split_USER/AHN3_T2.laz ...\n"
                    },
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Retiling: AHN3_T2.laz\n"
                    },
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:11,819 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:07:11,821 -                        laserfarm.retiler -       INFO - Setting up the target grid\n2023-11-14 14:07:11,823 -                        laserfarm.retiler -       INFO - Splitting file /tmp/AHN3_T2.laz_input/AHN3_T2.laz with PDAL ...\n2023-11-14 14:07:35,928 -                        laserfarm.retiler -       INFO - ... splitting completed.\n2023-11-14 14:07:35,934 -                        laserfarm.retiler -       INFO - Redistributing files to tiles ...\n2023-11-14 14:07:35,937 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_4.laz to tile_289_272\n2023-11-14 14:07:35,939 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_3.laz to tile_288_272\n2023-11-14 14:07:35,942 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_1.laz to tile_289_271\n2023-11-14 14:07:35,944 -                        laserfarm.retiler -       INFO - ... file AHN3_T2_2.laz to tile_288_271\n2023-11-14 14:07:35,945 -                        laserfarm.retiler -       INFO - ... redistributing completed.\n2023-11-14 14:07:35,947 -                        laserfarm.retiler -       INFO - Validating split ...\n2023-11-14 14:07:35,948 -                        laserfarm.retiler -       INFO - ... 13973048 points in parent file\n2023-11-14 14:07:35,950 -                        laserfarm.retiler -       INFO - ... 528208 points in AHN3_T2_4.laz\n2023-11-14 14:07:35,951 -                        laserfarm.retiler -       INFO - ... 9582849 points in AHN3_T2_2.laz\n2023-11-14 14:07:35,952 -                        laserfarm.retiler -       INFO - ... 1450445 points in AHN3_T2_1.laz\n2023-11-14 14:07:35,957 -                        laserfarm.retiler -       INFO - ... 2411546 points in AHN3_T2_3.laz\n2023-11-14 14:07:35,958 -                        laserfarm.retiler -       INFO - ... split validation completed.\n2023-11-14 14:07:35,960 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/retiled_USER ...\n2023-11-14 14:07:53,426 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:07:53,428 -           laserfarm.pipeline_remote_data -       INFO - Removing input and output folders\n2023-11-14 14:07:53,456 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/AHN3_T1.laz_input\n2023-11-14 14:07:53,458 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/AHN3_T1.laz_output\n2023-11-14 14:07:53,460 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/split_USER/AHN3_T1.laz ...\n"
                    },
                    {
                        "name": "stdout",
                        "output_type": "stream",
                        "text": "Retiling: AHN3_T1.laz\n"
                    },
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:07:58,926 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:07:58,928 -                        laserfarm.retiler -       INFO - Setting up the target grid\n2023-11-14 14:07:58,930 -                        laserfarm.retiler -       INFO - Splitting file /tmp/AHN3_T1.laz_input/AHN3_T1.laz with PDAL ...\n2023-11-14 14:08:44,639 -                        laserfarm.retiler -       INFO - ... splitting completed.\n2023-11-14 14:08:44,642 -                        laserfarm.retiler -       INFO - Redistributing files to tiles ...\n2023-11-14 14:08:44,645 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_3.laz to tile_289_271\n2023-11-14 14:08:44,647 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_2.laz to tile_289_270\n2023-11-14 14:08:44,649 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_4.laz to tile_288_271\n2023-11-14 14:08:44,651 -                        laserfarm.retiler -       INFO - ... file AHN3_T1_1.laz to tile_288_270\n2023-11-14 14:08:44,652 -                        laserfarm.retiler -       INFO - ... redistributing completed.\n2023-11-14 14:08:44,653 -                        laserfarm.retiler -       INFO - Validating split ...\n2023-11-14 14:08:44,654 -                        laserfarm.retiler -       INFO - ... 17822698 points in parent file\n2023-11-14 14:08:44,655 -                        laserfarm.retiler -       INFO - ... 2563278 points in AHN3_T1_4.laz\n2023-11-14 14:08:44,656 -                        laserfarm.retiler -       INFO - ... 603139 points in AHN3_T1_3.laz\n2023-11-14 14:08:44,658 -                        laserfarm.retiler -       INFO - ... 12765152 points in AHN3_T1_1.laz\n2023-11-14 14:08:44,659 -                        laserfarm.retiler -       INFO - ... 1891129 points in AHN3_T1_2.laz\n2023-11-14 14:08:44,660 -                        laserfarm.retiler -       INFO - ... split validation completed.\n2023-11-14 14:08:44,661 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/retiled_USER ...\n2023-11-14 14:09:02,425 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:09:02,427 -           laserfarm.pipeline_remote_data -       INFO - Removing input and output folders\n"
                    }
                ],
                "id": "d33b3708-c431-40a5-9e69-10b3c7777e85"
            },
            {
                "cell_type": "markdown",
                "source": "## Normalization",
                "metadata": {},
                "id": "cf7fdf79-530d-41a7-88c3-1177d1a088b2"
            },
            {
                "cell_type": "code",
                "source": "# S5 Fetch Tiles for Norm\nremote_path_norm\n\ntiles = [t.strip('/') for t in list_remote(get_wdclient(conf_wd_opts), conf_remote_path_retiled.as_posix())\n         if fnmatch.fnmatch(t, 'tile_*_*/')]",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 7,
                "outputs": [],
                "id": "8e8cfd3a-288d-43f2-9f69-298ec0b28751"
            },
            {
                "cell_type": "code",
                "source": "# S6 Normalization\n\nimport copy\n\ntiles\n\nremote_path_norm = str(conf_remote_path_norm)\n\nnormalization_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_retiled.as_posix(),\n    'load': {'attributes': 'all'},\n    # Filter out artifically high points - give overflow error when writing\n    'apply_filter': {'filter_type':'select_below',\n                     'attribute': 'z',\n                     'threshold': 10000.},  # remove non-physically heigh points\n    'normalize': 1,\n    'clear_cache' : {},\n    'pushremote': conf_remote_path_norm.as_posix(),\n}\n\n# write input dictionary to JSON file\nwith open('normalize.json', 'w') as f:\n    json.dump(normalization_input, f)\n    \n\n# add pipeline list to macro-pipeline object and set the corresponding labels\n#tile = tiles\n# for tile in tiles:\n#normalization_input_ = copy.deepcopy(normalization_input)\n#normalization_input_['export_point_cloud'] = {'filename': '{}.laz'.format(tile),'overwrite': True}\n#dp = DataProcessing(tile, label=tile).config(normalization_input_).setup_webdav_client(conf_wd_opts)\n\nfor tile in tiles:\n    normalization_input_ = copy.deepcopy(normalization_input)\n    normalization_input_['export_point_cloud'] = {'filename': '{}.laz'.format(tile),'overwrite': True}\n    dp = DataProcessing(tile, label=tile).config(normalization_input_).setup_webdav_client(conf_wd_opts)\n    dp.run()\n\nremote_path_norm\n",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 8,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:09:03,043 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_271_input\n2023-11-14 14:09:03,045 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_271_output\n2023-11-14 14:09:03,049 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_271 ...\n2023-11-14 14:09:07,069 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:09:07,071 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:09:07,072 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_271_input/tile_289_271/AHN3_T1_3.laz\n2023-11-14 14:09:07,478 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_271_input/tile_289_271/AHN3_T2_1.laz\n2023-11-14 14:09:20,918 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:09:20,919 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:09:21,164 -                                     root -       INFO - Cylinder size in Bytes: 692155693.4389035\n2023-11-14 14:09:21,165 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:09:21,167 -                                     root -       INFO - Start tree creation\n2023-11-14 14:09:25,841 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:09:25,855 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:09:34,368 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:09:34,370 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:09:46,418 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:09:46,421 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_271_output/tile_289_271.laz\n2023-11-14 14:10:11,878 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:10:11,883 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:10:11,886 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:10:17,603 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:10:17,610 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_271_input\n2023-11-14 14:10:17,612 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_271_output\n2023-11-14 14:10:17,613 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_271 ...\n2023-11-14 14:10:23,807 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:10:23,810 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:10:23,811 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_271_input/tile_288_271/AHN3_T1_4.laz\n2023-11-14 14:10:25,596 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_271_input/tile_288_271/AHN3_T2_2.laz\n2023-11-14 14:11:06,236 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:11:06,238 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:11:07,911 -                                     root -       INFO - Cylinder size in Bytes: 5723102168.897599\n2023-11-14 14:11:07,913 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:11:07,914 -                                     root -       INFO - Start tree creation\n2023-11-14 14:11:10,516 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:11:10,623 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:12:09,087 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:12:09,088 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:12:10,448 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:12:10,449 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_271_output/tile_288_271.laz\n2023-11-14 14:12:21,933 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:12:21,934 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:12:21,957 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:12:33,642 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:12:33,663 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_272_input\n2023-11-14 14:12:33,665 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_272_output\n2023-11-14 14:12:33,668 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_272 ...\n2023-11-14 14:12:36,571 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:12:36,572 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:12:36,573 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_272_input/tile_288_272/AHN3_T2_3.laz\n2023-11-14 14:12:38,182 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:12:38,184 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:12:38,620 -                                     root -       INFO - Cylinder size in Bytes: 1241913170.650779\n2023-11-14 14:12:38,622 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:12:38,624 -                                     root -       INFO - Start tree creation\n2023-11-14 14:12:39,033 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:12:39,061 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:12:50,971 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:12:50,974 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:12:51,062 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:12:51,064 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_272_output/tile_288_272.laz\n2023-11-14 14:12:53,077 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:12:53,078 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:12:53,081 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:12:57,999 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:12:58,010 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_272_input\n2023-11-14 14:12:58,011 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_272_output\n2023-11-14 14:12:58,013 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_272 ...\n2023-11-14 14:13:00,562 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:13:00,564 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:13:00,565 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_272_input/tile_289_272/AHN3_T2_4.laz\n2023-11-14 14:13:00,921 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:13:00,922 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:13:00,973 -                                     root -       INFO - Cylinder size in Bytes: 150197785.47624204\n2023-11-14 14:13:00,974 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:13:00,975 -                                     root -       INFO - Start tree creation\n2023-11-14 14:13:01,066 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:13:01,070 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:13:03,304 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:13:03,306 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:13:03,317 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:13:03,318 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_272_output/tile_289_272.laz\n2023-11-14 14:13:03,726 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:13:03,728 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:13:03,728 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:13:07,560 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:13:07,568 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_288_270_input\n2023-11-14 14:13:07,569 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_288_270_output\n2023-11-14 14:13:07,572 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_288_270 ...\n2023-11-14 14:13:12,688 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:13:12,691 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:13:12,692 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_288_270_input/tile_288_270/AHN3_T1_1.laz\n2023-11-14 14:13:22,049 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:13:22,050 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:13:23,300 -                                     root -       INFO - Cylinder size in Bytes: 4486912100.415718\n2023-11-14 14:13:23,301 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:13:23,302 -                                     root -       INFO - Start tree creation\n2023-11-14 14:13:25,700 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:13:25,795 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:14:21,239 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:14:21,243 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:14:22,442 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:14:22,448 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_288_270_output/tile_288_270.laz\n2023-11-14 14:14:34,067 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:14:34,068 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:14:34,084 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:14:46,022 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:14:46,057 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_289_270_input\n2023-11-14 14:14:46,059 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_289_270_output\n2023-11-14 14:14:46,061 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/retiled_USER/tile_289_270 ...\n2023-11-14 14:14:48,934 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:14:48,937 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:14:48,938 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_289_270_input/tile_289_270/AHN3_T1_2.laz\n2023-11-14 14:14:50,295 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:14:50,297 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:14:50,452 -                                     root -       INFO - Cylinder size in Bytes: 542650063.6561003\n2023-11-14 14:14:50,453 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:14:50,454 -                                     root -       INFO - Start tree creation\n2023-11-14 14:14:50,790 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:14:50,803 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:14:58,704 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:14:58,708 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:14:58,798 -                laserfarm.data_processing -       INFO - Exporting environment point-cloud ...\n2023-11-14 14:14:58,799 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_289_270_output/tile_289_270.laz\n2023-11-14 14:15:00,478 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:15:00,480 -                laserfarm.data_processing -       INFO - Clearing cached KDTrees ...\n2023-11-14 14:15:00,483 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/norm_USER ...\n2023-11-14 14:15:05,681 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n"
                    },
                    {
                        "execution_count": 8,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "'/webdav/vl-laserfarm/norm_USER'"
                        },
                        "metadata": {}
                    }
                ],
                "id": "365ab2e8-c194-4828-90cd-7d91fe815b58"
            },
            {
                "cell_type": "markdown",
                "source": "## Feature extraction",
                "metadata": {},
                "id": "caa87bfe-cfe8-4aea-a292-e4d11e479dd4"
            },
            {
                "cell_type": "code",
                "source": "# S7 Fetch Norm Tiles for feature\nremote_path_norm\n\ntiles = [f for f in list_remote(get_wdclient(conf_wd_opts), pathlib.Path(conf_remote_path_norm).as_posix())\n             if f.lower().endswith('.laz')]\n\ntiles",
                "metadata": {
                    "trusted": true
                },
                "execution_count": 9,
                "outputs": [
                    {
                        "execution_count": 9,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "['tile_289_271.laz',\n 'tile_288_271.laz',\n 'tile_288_272.laz',\n 'tile_289_272.laz',\n 'tile_288_270.laz',\n 'tile_289_270.laz']"
                        },
                        "metadata": {}
                    }
                ],
                "id": "71329832-64e6-406d-9f03-e1f013754308"
            },
            {
                "cell_type": "code",
                "source": "# S8 Feature Extraction\n\nremote_path_targets = str(conf_remote_path_targets)\n\nfeatures = [\"perc_95_normalized_height\"]\n\ntile_mesh_size = float(conf_tile_mesh_size)\n\ngrid_feature = {\n    'min_x': float(conf_min_x),\n    'max_x': float(conf_max_x),\n    'min_y': float(conf_min_y),\n    'max_y': float(conf_max_y),\n    'n_tiles_side': int(conf_n_tiles_side)\n}\n\nfeature_extraction_input = {\n    'setup_local_fs': {\n        'input_folder': (conf_local_tmp / 'tile_input').as_posix(),\n        'output_folder': (conf_local_tmp / 'tile_output').as_posix(),\n    },\n    'pullremote': conf_remote_path_norm.as_posix(),\n    'load': {'attributes': [conf_attribute]},\n    'normalize': 1,\n    'apply_filter': {\n        'filter_type': conf_filter_type, \n        'attribute': conf_attribute,\n        'value': [int(conf_apply_filter_value)]#ground surface (2), water (9), buildings (6), artificial objects (26), vegetation (?), and unclassified (1)\n    },\n    'generate_targets': {\n        'tile_mesh_size' : tile_mesh_size,\n        'validate' : True,\n        'validate_precision': float(conf_validate_precision),\n        **grid_feature\n    },\n    'extract_features': {\n        'feature_names': features,\n        'volume_type': 'cell',\n        'volume_size': tile_mesh_size\n    },\n    'export_targets': {\n        'attributes': features,\n        'multi_band_files': False\n    },\n    'pushremote': conf_remote_path_targets.as_posix(),\n#     'cleanlocalfs': {}\n}\n\nfor t in tiles:    \n    idx = (t.split('.')[0].split('_')[1:])\n    processing = DataProcessing(t, tile_index=idx,label=t).config(feature_extraction_input).setup_webdav_client(conf_wd_opts)\n    processing.run()\n\nremote_path_targets",
                "metadata": {
                    "tags": [],
                    "trusted": true
                },
                "execution_count": 10,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:15:06,342 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:15:06,343 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:15:06,344 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_271.laz ...\n2023-11-14 14:15:08,991 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:15:08,993 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:15:08,994 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_271.laz\n2023-11-14 14:15:11,073 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:15:11,074 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:15:11,285 -                                     root -       INFO - Cylinder size in Bytes: 692155693.4389035\n2023-11-14 14:15:11,287 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:11,292 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:11,674 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:11,691 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:15:20,371 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:15:20,373 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:15:20,431 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:15:20,433 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,271)\n2023-11-14 14:15:20,452 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:15:20,455 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:15:20,456 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:15:20,457 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:15:20,460 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:15:20,461 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:20,462 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:20,607 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:20,610 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:15:22,663 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:15:22,808 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.14 seconds\n2023-11-14 14:15:22,816 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:15:22,839 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:15:22,841 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_271.ply\n2023-11-14 14:15:22,959 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:15:22,960 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:15:27,518 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:15:27,527 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:15:27,528 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:15:27,530 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_271.laz ...\n2023-11-14 14:15:34,971 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:15:34,973 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:15:34,973 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_271.laz\n2023-11-14 14:15:47,228 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:15:47,230 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:15:48,887 -                                     root -       INFO - Cylinder size in Bytes: 5723102168.897599\n2023-11-14 14:15:48,889 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:15:48,891 -                                     root -       INFO - Start tree creation\n2023-11-14 14:15:51,042 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:15:51,155 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:16:49,085 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:16:49,088 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:16:49,337 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:16:49,339 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,271)\n2023-11-14 14:16:49,417 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:16:49,420 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:16:49,421 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:16:49,422 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:16:49,425 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:16:49,426 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:16:49,427 -                                     root -       INFO - Start tree creation\n2023-11-14 14:16:49,962 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:16:49,966 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:16:56,213 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:16:56,681 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.47 seconds\n2023-11-14 14:16:56,703 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:16:56,755 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:16:56,757 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_271.ply\n2023-11-14 14:16:56,888 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:16:56,889 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:03,871 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:03,886 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:03,888 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:03,891 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_272.laz ...\n2023-11-14 14:17:06,555 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:06,559 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:06,560 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_272.laz\n2023-11-14 14:17:08,896 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:17:08,898 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:17:09,252 -                                     root -       INFO - Cylinder size in Bytes: 1241913170.650779\n2023-11-14 14:17:09,254 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:09,255 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:09,660 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:09,686 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:21,395 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:17:21,397 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:17:21,424 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:17:21,425 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,272)\n2023-11-14 14:17:21,431 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:17:21,433 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:17:21,434 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:17:21,436 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:17:21,437 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:17:21,438 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:21,439 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:21,477 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:21,480 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:21,977 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:17:22,041 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.06 seconds\n2023-11-14 14:17:22,044 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:17:22,049 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:17:22,051 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_272.ply\n2023-11-14 14:17:22,170 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:17:22,171 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:31,860 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:31,868 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:31,869 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:31,872 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_272.laz ...\n2023-11-14 14:17:33,660 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:33,662 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:33,663 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_272.laz\n2023-11-14 14:17:34,209 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:17:34,211 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:17:34,255 -                                     root -       INFO - Cylinder size in Bytes: 150197785.47624204\n2023-11-14 14:17:34,256 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:34,257 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:34,344 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:34,348 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:36,287 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:17:36,289 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:17:36,310 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:17:36,312 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,272)\n2023-11-14 14:17:36,316 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:17:36,317 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:17:36,318 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:17:36,319 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:17:36,320 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:17:36,321 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:17:36,322 -                                     root -       INFO - Start tree creation\n2023-11-14 14:17:36,380 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:17:36,384 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:17:37,099 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:17:37,160 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.06 seconds\n2023-11-14 14:17:37,163 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:17:37,171 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:17:37,172 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_272.ply\n2023-11-14 14:17:37,297 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:17:37,298 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:17:49,909 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:17:49,916 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:17:49,917 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:17:49,921 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_288_270.laz ...\n2023-11-14 14:17:58,019 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:17:58,021 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:17:58,022 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_288_270.laz\n2023-11-14 14:18:12,472 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:18:12,478 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:18:13,831 -                                     root -       INFO - Cylinder size in Bytes: 4486912100.415718\n2023-11-14 14:18:13,833 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:18:13,834 -                                     root -       INFO - Start tree creation\n2023-11-14 14:18:16,697 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:18:16,783 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:11,409 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:19:11,413 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:19:12,098 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:19:12,101 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (288,270)\n2023-11-14 14:19:12,213 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:19:12,215 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:19:12,216 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:19:12,217 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:19:12,220 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:19:12,221 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:12,222 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:12,929 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:12,933 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:21,551 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:19:22,165 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.61 seconds\n2023-11-14 14:19:22,195 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:19:22,271 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:19:22,274 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_288_270.ply\n2023-11-14 14:19:22,419 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:19:22,420 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:19:39,549 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n2023-11-14 14:19:39,566 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/tile_input\n2023-11-14 14:19:39,567 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/tile_output\n2023-11-14 14:19:39,576 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/norm_USER/tile_289_270.laz ...\n2023-11-14 14:19:42,000 -           laserfarm.pipeline_remote_data -       INFO - ... pulling completed.\n2023-11-14 14:19:42,001 -                laserfarm.data_processing -       INFO - Loading point cloud data ...\n2023-11-14 14:19:42,004 -                laserfarm.data_processing -       INFO - ... loading /tmp/tile_input/tile_289_270.laz\n2023-11-14 14:19:43,821 -                laserfarm.data_processing -       INFO - ... loading completed.\n2023-11-14 14:19:43,852 -                laserfarm.data_processing -       INFO - Normalizing point-cloud heights ...\n2023-11-14 14:19:44,008 -                                     root -       INFO - Cylinder size in Bytes: 542650063.6561003\n2023-11-14 14:19:44,009 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:44,011 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:44,333 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:44,346 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:51,518 -                laserfarm.data_processing -       INFO - ... normalization completed.\n2023-11-14 14:19:51,520 -                laserfarm.data_processing -       INFO - Filtering point-cloud data\n2023-11-14 14:19:51,553 -                laserfarm.data_processing -       INFO - Setting up the target grid\n2023-11-14 14:19:51,555 -                laserfarm.data_processing -       INFO - Checking whether points belong to cell (289,270)\n2023-11-14 14:19:51,564 -                laserfarm.data_processing -       INFO - Generating target point mesh with 10.0m spacing \n2023-11-14 14:19:51,566 -                laserfarm.data_processing -       INFO - Building volume of type cell\n2023-11-14 14:19:51,568 -                laserfarm.data_processing -       INFO - Constructing neighborhoods\n2023-11-14 14:19:51,569 -                laserfarm.data_processing -       INFO - Starting feature extraction ...\n2023-11-14 14:19:51,571 -                                     root -       INFO - Cylinder size in Bytes: 6408849013.323179\n2023-11-14 14:19:51,572 -                                     root -       INFO - Memory size in Bytes: 15344021504\n2023-11-14 14:19:51,572 -                                     root -       INFO - Start tree creation\n2023-11-14 14:19:51,646 -                                     root -       INFO - Done with env tree creation\n2023-11-14 14:19:51,650 -                                     root -       INFO - Done with target tree creation\n2023-11-14 14:19:52,853 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\"\n2023-11-14 14:19:52,968 -                                     root -       INFO - Extracting feature(s) \"['perc_95_normalized_height']\" took 0.11 seconds\n2023-11-14 14:19:52,974 -                laserfarm.data_processing -       INFO - ... feature extraction completed.\n2023-11-14 14:19:52,985 -                laserfarm.data_processing -       INFO - Exporting target point-cloud ...\n2023-11-14 14:19:52,987 -                laserfarm.data_processing -       INFO - ... exporting /tmp/tile_output/perc_95_normalized_height/tile_289_270.ply\n2023-11-14 14:19:53,120 -                laserfarm.data_processing -       INFO - ... exporting completed.\n2023-11-14 14:19:53,121 -           laserfarm.pipeline_remote_data -       INFO - Pushing to WebDAV /webdav/vl-laserfarm/targets_USER ...\n2023-11-14 14:20:13,366 -           laserfarm.pipeline_remote_data -       INFO - ... pushing completed.\n"
                    },
                    {
                        "execution_count": 10,
                        "output_type": "execute_result",
                        "data": {
                            "text/plain": "'/webdav/vl-laserfarm/targets_USER'"
                        },
                        "metadata": {}
                    }
                ],
                "id": "91bbaa1c-9d83-4703-9477-dcd88522ded2"
            },
            {
                "cell_type": "markdown",
                "source": "## GeoTIFF export",
                "metadata": {
                    "tags": []
                },
                "id": "2eadeb04-efe1-4b68-810b-efd4a91b8c18"
            },
            {
                "cell_type": "code",
                "source": "# S9 GeoTIFF Export\n\nremote_path_targets\n\nremote_path_geotiffs = conf_remote_path_geotiffs\n\n# setup input dictionary to configure the GeoTIFF export pipeline\ngeotiff_export_input = {\n    'setup_local_fs': {'tmp_folder': conf_local_tmp.as_posix()},\n    'pullremote': conf_remote_path_targets.as_posix(),\n    'parse_point_cloud': {},\n    'data_split': {'xSub': 1, 'ySub': 1},\n    'create_subregion_geotiffs': {'output_handle': 'geotiff'},\n    'pushremote': remote_path_geotiffs.as_posix(),\n    'cleanlocalfs': {}   \n}\n\nwriter = GeotiffWriter(input_dir=conf_feature_name, bands=conf_feature_name,label=conf_feature_name).config(geotiff_export_input).setup_webdav_client(conf_wd_opts)\nwriter.run()",
                "metadata": {
                    "trusted": true
                },
                "execution_count": null,
                "outputs": [
                    {
                        "name": "stderr",
                        "output_type": "stream",
                        "text": "2023-11-14 14:20:13,387 -           laserfarm.pipeline_remote_data -       INFO - Input dir set to /tmp/perc_95_normalized_height_input\n2023-11-14 14:20:13,388 -           laserfarm.pipeline_remote_data -       INFO - Output dir set to /tmp/perc_95_normalized_height_output\n2023-11-14 14:20:13,391 -           laserfarm.pipeline_remote_data -       INFO - Pulling from WebDAV /webdav/vl-laserfarm/targets_USER/perc_95_normalized_height ...\n"
                    }
                ],
                "id": "cdd5ac0b-78e7-40c8-81ff-0078d3916ecd"
            },
            {
                "cell_type": "code",
                "source": "",
                "metadata": {
                    "trusted": true
                },
                "execution_count": null,
                "outputs": [],
                "id": "08ea3b0b-8627-44ab-a8da-56d7ce531e7e"
            }
        ]
    }
[I 2023-11-16 10:06:52.621 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/extract?1700129212416 (anonymous@10.244.4.1) 97.74ms
[I 2023-11-16 10:06:54.484 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129214445 (anonymous@10.244.0.0) 2.31ms
[I 2023-11-16 10:06:54.529 ServerApp] 200 GET /vl-laserfarm/user/USER/api/terminals?1700129214461 (USER@10.244.7.0) 2.39ms
[I 2023-11-16 10:06:54.756 ServerApp] 200 GET /vl-laserfarm/user/USER/api/contents?content=1&1700129214711 (USER@10.244.9.0) 23.74ms
[I 2023-11-16 10:06:55.760 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129215732 (anonymous@10.244.11.0) 2.43ms
[I 2023-11-16 10:06:56.569 ServerApp] 200 GET /vl-laserfarm/user/USER/api/sessions?1700129216528 (USER@10.244.4.1) 2.27ms
[I 2023-11-16 10:06:56.664 ServerApp] 200 GET /vl-laserfarm/user/USER/api/kernels?1700129216622 (USER@10.244.0.0) 2.56ms
[I 2023-11-16 10:06:57.441 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129217405 (anonymous@10.244.7.0) 1.76ms
[I 2023-11-16 10:06:58.784 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129218742 (anonymous@10.244.9.0) 1.89ms
[I 2023-11-16 10:07:00.574 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129220546 (anonymous@10.244.11.0) 1.27ms
[I 2023-11-16 10:07:02.007 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/types?1700129221976 (anonymous@10.244.4.1) 1.88ms
[I 2023-11-16 10:07:04.390 ServerApp] 200 POST /vl-laserfarm/user/USER/vre/containerizer/baseimage?1700129224344 (anonymous@10.244.0.0) 1.23ms
[I 2023-11-16 10:07:04.590 ServerApp] 200 GET /vl-laserfarm/user/USER/api/terminals?1700129224556 (USER@10.244.7.0) 2.03ms
[I 2023-11-16 10:07:04.833 ServerApp] 200 GET /vl-laserfarm/user/USER/api/contents?content=1&1700129224771 (USER@10.244.9.0) 30.76ms
[E 2023-11-16 10:07:05.737 ServerApp] Uncaught exception POST /vl-laserfarm/user/USER/vre/containerizer/addcell?1700129225295 (10.244.11.0)
    HTTPServerRequest(protocol='https', host='lfw-ds001-i040.lifewatch.dev', method='POST', uri='/vl-laserfarm/user/USER/vre/containerizer/addcell?1700129225295', version='HTTP/1.1', remote_ip='10.244.11.0')
    Traceback (most recent call last):
      File "/venv/lib/python3.9/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
      File "/venv/lib/python3.9/site-packages/jupyterlab_vre/component_containerizer/handlers.py", line 284, in post
        build_templates(cell=current_cell, files_info=files_info)
      File "/venv/lib/python3.9/site-packages/jupyterlab_vre/component_containerizer/handlers.py", line 522, in build_templates
        compiled_code = autopep8.fix_code(compiled_code)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 3506, in fix_code
        return fix_lines(sio.readlines(), options=options)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 3569, in fix_lines
        fixed_source = fix.fix()
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 613, in fix
        self._fix_source(filter_results(source=''.join(self.source),
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 557, in _fix_source
        modified_lines = fix(result)
      File "/venv/lib/python3.9/site-packages/autopep8.py", line 761, in fix_e225
        pycodestyle.missing_whitespace_around_operator(fixed, ts))
    AttributeError: module 'pycodestyle' has no attribute 'missing_whitespace_around_operator'
[W 2023-11-16 10:07:05.879 ServerApp] Unhandled error
[E 2023-11-16 10:07:05.881 ServerApp] {
      "Cookie": "jupyterhub-user-USER=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]; csrftoken=[secret]; __Host-next-auth.csrf-token=[secret]; __Secure-next-auth.callback-url=[secret]; __Secure-next-auth.session-token=[secret]",
      "Cache-Control": "no-cache",
      "Pragma": "no-cache",
      "Sec-Gpc": "1",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Dnt": "1",
      "Origin": "https://lfw-ds001-i040.lifewatch.dev",
      "X-Xsrftoken": "2|4ed1ec1a|f6bdfda286310c3729034ccdae5667dc|1699354541",
      "Authorization": "token [secret]",
      "Content-Type": "text/plain;charset=UTF-8",
      "Referer": "https://lfw-ds001-i040.lifewatch.dev/vl-laserfarm/user/USER/lab?",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "*/*",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0",
      "Content-Length": "20",
      "X-Scheme": "https",
      "X-Forwarded-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "lfw-ds001-i040.lifewatch.dev",
      "X-Forwarded-For": "10.244.11.0,::ffff:10.244.4.89",
      "X-Real-Ip": "10.244.11.0",
      "X-Request-Id": "b912fcf086e521477cc097f1d0aad919",
      "Host": "lfw-ds001-i040.lifewatch.dev",
      "Connection": "close"
    }
[E 2023-11-16 10:07:05.881 ServerApp] 500 POST /vl-laserfarm/user/USER/vre/containerizer/addcell?1700129225295 (anonymous@10.244.11.0) 564.08ms
skoulouzis commented 11 months ago

The line causing the error is :

'filter_type': conf_filter_type