search

emplexed / vgi-analytics-framework

Analytics framework for historic VGI datasets
Apache License 2.0
8 stars 1 forks source link

Unknown Error #2

Closed ktjaco closed 7 years ago

ktjaco commented 7 years ago

Hi,

I've successfully run the -importer and -quadtree commands but I am receiving this unknown error when trying to run an analysis. I have my settings XML specified and what I thought to be the proper syntax. The -p setting is optional but where would I find the polygon raster file?

$ java -Xmx12g -jar vgi-analytics-framework-0.1.jar -analysis -s ../settings/settings_op_generator.xml

log4j:WARN Continuable parsing error 24 and column 15
log4j:WARN Element type "log4j:configuration" must be declared.
log4j:WARN Continuable parsing error 29 and column 68
log4j:WARN Element type "appender" must be declared.
log4j:WARN Continuable parsing error 30 and column 73
log4j:WARN Element type "errorHandler" must be declared.
log4j:WARN Continuable parsing error 32 and column 45
log4j:WARN Element type "param" must be declared.
log4j:WARN Continuable parsing error 33 and column 43
log4j:WARN Element type "param" must be declared.
log4j:WARN Continuable parsing error 34 and column 50
log4j:WARN Element type "layout" must be declared.
log4j:WARN Continuable parsing error 35 and column 87
log4j:WARN Element type "param" must be declared.
log4j:WARN Continuable parsing error 40 and column 8
log4j:WARN Element type "root" must be declared.
log4j:WARN Continuable parsing error 41 and column 28
log4j:WARN Element type "priority" must be declared.
log4j:WARN Continuable parsing error 42 and column 33
log4j:WARN Element type "appender-ref" must be declared.
log4j: reset attribute= "".
log4j: Threshold ="".
log4j: Level value for root is  [INFO].
log4j: root level set to INFO
log4j: Class name: [org.apache.log4j.ConsoleAppender]
log4j: Setting property [target] to [System.out].
log4j: Setting property [threshold] to [DEBUG].
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d{HH:mm:ss,SSS} %-5p [%C{1}.%M()]  %m%n].
log4j: Adding appender named [CONSOLE] to category [root].
11:47:10,421 INFO  [AbstractApplicationContext.prepareRefresh()]  Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@ba8a1dc: startup date [Wed Mar 08 11:47:10 EST 2017]; root of context hierarchy
11:47:10,454 INFO  [XmlBeanDefinitionReader.loadBeanDefinitions()]  Loading XML bean definitions from class path resource [application-context-vgi-pipeline.xml]
11:47:11,159 INFO  [VgiPipelineSettings.loadSettings()]  Settings 'Op Generator' loaded
11:47:11,159 INFO  [VgiAnalysis.launch()]  1 filter polygons found!
11:47:11,159 INFO  [VgiAnalysis.launch()]  Start analysis... Austria
11:47:11,167 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
11:47:11,167 INFO  [AbstractApplicationContext.doClose()]  Closing org.springframework.context.support.ClassPathXmlApplicationContext@ba8a1dc: startup date [Wed Mar 08 11:47:10 EST 2017]; root of context hierarchy
11:47:11,168 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
11:47:11,172 ERROR [VgiAnalysis.launch()]  Unknown error
java.lang.NullPointerException
    at at.salzburgresearch.vgi.vgianalyticsframework.activityanalysis.pipeline.consumer.impl.VgiAnalysisConsumer.doBeforeFirstBatch(VgiAnalysisConsumer.java:105)
    at at.salzburgresearch.vgi.vgianalyticsframework.activityanalysis.pipeline.impl.VgiPipelineImpl.doBeforeFirstBatch(VgiPipelineImpl.java:167)
    at at.salzburgresearch.vgi.vgianalyticsframework.activityanalysis.pipeline.impl.VgiPipelineImpl.start(VgiPipelineImpl.java:121)
    at at.salzburgresearch.vgi.vgianalyticsframework.activityanalysis.application.VgiAnalysis.launch(VgiAnalysis.java:145)
    at at.salzburgresearch.vgi.vgianalyticsframework.activityanalysis.application.VgiAnalyticsFramework.main(VgiAnalyticsFramework.java:50)
usage: java -jar vgi-activity-1.0.jar [OPTION]...
 -analysis,--vgi_analysis               VGI analysis
 -c,--crs <crs>                         crs
 -d,--directory <home_directory_file>   home directory file
 -ea,--east <east_bound>                east bound
 -h,--help                              Display this help page
 -importer,--osm_importer               OSM Importer
 -no,--north <north_bound>              north bound
 -o,--osm <osm_history_file>            osm history file
 -p,--polygons <polygon_file>           polygon file
 -quadtree,--quadtree_builder           Quadtree Builder
 -r,--rastersize <raster_size>          raster size
 -raster,--hexagonal_raster_generator   Hexagonal raster generator
 -s,--settings <settings_file>          settings file
 -so,--south <south_bound>              south bound
 -we,--west <west_bound>                west bound
SGroe commented 7 years ago

You are using the settings file for the operation generator. This file doesn't specify analysis methods, feature types, ... Try to use a settings file which I prepared for analyzing features (e.g. settings.xml)

I will implement a friendlier error message.

Thanks for giving this tool a try.

ktjaco commented 7 years ago

@SGroe I think it is a very useful tool. Once I get a functional workflow, I can offer to write some documentation for the exact commands to use in the terminal.

ktjaco commented 7 years ago

@SGroe Here is a look at my settings.xml file. Is this the correct command to run the -importer for the spatial filter for Austria? If so, how long should this import take?

java -Xmx14g -jar vgi-analytics-framework-0.1.jar -importer -s ../settings/settings.xml -o ../pbf/history-latest.osm.pbf filterPolygon

<?xml version="1.0" encoding="UTF-8" ?>
<settings
        xmlns="http://www.salzburgresearch.at"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.salzburgresearch.at validator.xsd">
    <general
        settingName="GeneralAnalysis"
        pbfDataFolder="/home/user/vgi-analytics-framework/pbf"
        readQuadtree="false"
        resultFolder="/home/user/vgi-analytics-framework/results"
        filterTimestamp="2020-01-01T00:00:00Z" />

    <filterTags>
        <includeTag key="place" value="" />
        <includeTag key="highway" value="crossing" />
        <includeTag key="natural" value="" />
        <includeTag key="building" value="" />
        <includeTag key="damage" value="" />
        <includeTag key="damage:event" value="" />
        <includeTag key="collapsed" value="" />
        <includeTag key="condition" value="" />
        <includeTag key="physical:condition" value="" />
        <includeTag key="amenity" value="" />
        <includeTag key="natural" value="_value" />
        <includeTag key="landuse" value="_value" />
        <includeTag key="highway" value="motorway_junction" />
        <includeTag key="highway" value="motorway" />
        <includeTag key="highway" value="motorway_link" />
        <includeTag key="highway" value="trunk" />
        <includeTag key="highway" value="trunk_link" />
        <includeTag key="highway" value="primary" />
        <includeTag key="highway" value="primary_link" />
        <includeTag key="highway" value="secondary" />
        <includeTag key="highway" value="secondary_link" />
        <includeTag key="highway" value="tertiary" />
        <includeTag key="highway" value="tertiary_link" />
        <includeTag key="highway" value="residential" />
        <includeTag key="highway" value="unclassified" />
        <includeTag key="highway" value="living_street" />
        <includeTag key="highway" value="service" />
        <includeTag key="highway" value="track" />
        <includeTag key="highway" value="road" />
        <includeTag key="highway" value="" /><includeTag key="building" value="" /><includeTag key="amenity" value="" /><includeTag key="landuse" value="" />
        <includeTag key="aerialway" value="" /><includeTag key="aeroway" value="" /><includeTag key="barrier" value="" /><includeTag key="boundary" value="" />
        <includeTag key="craft" value="" /><includeTag key="emergency" value="" /><includeTag key="geological" value="" /><includeTag key="historic" value="" />
        <includeTag key="leisure" value="" /><includeTag key="man_made" value="" /><includeTag key="military" value="" /><includeTag key="natural" value="" />
        <includeTag key="office" value="" /><includeTag key="place" value="" /><includeTag key="power" value="" /><includeTag key="public_transport" value="" />
        <includeTag key="railway" value="" /><includeTag key="route" value="" /><includeTag key="shop" value="" /><includeTag key="sport" value="" />
        <includeTag key="tourism" value="" /><includeTag key="waterway" value="" />
    </filterTags>

    <!-- temporalResolution: eternity, century, decade, year, month, day -->
    <!-- findRelatedOperations e.g. OP_SPLIT_WAY, OP_MERGE_WAY; very time-consuming -->
    <!-- actionTimeBuffer: 1000*60*60*24=86400000=1 day ; 1000*60*60*12=43200000=12hours -->
    <analysis
        filterElementType="UNDEFINED"
        analysisStartDate="2004-01-01T00:00:00Z"
        analysisEndDate="2016-01-01T00:00:00Z"
        temporalResolution="month"
        findRelatedOperations="false"
        writeGeometryFiles="true"
        actionTimeBuffer="43200000">

        <featureType name="street" geometryType="LineString">
            <includeTag key="highway" value="motorway" />
            <includeTag key="highway" value="motorway_link" />
            <includeTag key="highway" value="trunk" />
            <includeTag key="highway" value="trunk_link" />
            <includeTag key="highway" value="primary" />
            <includeTag key="highway" value="primary_link" />
            <includeTag key="highway" value="secondary" />
            <includeTag key="highway" value="secondary_link" />
            <includeTag key="highway" value="tertiary" />
            <includeTag key="highway" value="tertiary_link" />
            <includeTag key="highway" value="residential" />
            <includeTag key="highway" value="unclassified" />
            <includeTag key="highway" value="living_street" />
            <includeTag key="highway" value="service" />
            <includeTag key="highway" value="track" />
            <includeTag key="highway" value="road" />
            <property key="highway" dataType="String" />
            <property key="name" dataType="String" />
            <property key="ref" dataType="String" />
            <property key="maxspeed" dataType="String" />
            <property key="bridge" dataType="String" />
            <property key="tunnel" dataType="String" />
            <property key="layer" dataType="String" />
        </featureType>
        <!--featureType name="street" geometryType="Point">
            <includeTag key="highway" value="" />
            <property key="highway" dataType="String" />
            <property key="name" dataType="String" />
            <property key="ref" dataType="String" />
            <property key="maxspeed" dataType="String" />
            <property key="bridge" dataType="String" />
            <property key="tunnel" dataType="String" />
            <property key="layer" dataType="String" />
        </featureType-->
        <featureType name="natural" geometryType="Point">
            <includeTag key="natural" value="" />
            <property key="natural" dataType="String" />
        </featureType>
        <featureType name="building" geometryType="Polygon">
            <includeTag key="building" value="" />
            <property key="building" />
            <property key="name" />
            <property key="damage" />
            <property key="damage:event" />
            <property key="collapsed" />
            <property key="condition" />
            <property key="physical:condition" />
        </featureType>
        <featureType name="place" geometryType="Point">
            <includeTag key="place" value="" />
            <property key="place" />
            <property key="name" />
        </featureType>
        <featureType name="housenumber" geometryType="Point">
            <includeTag key="addr:housenumber" value="" />
            <property key="addr:housenumber" />
            <property key="addr:street" />
        </featureType>
        <featureType name="amenity" geometryType="Point">
            <includeTag key="amenity" value="" />
            <property key="amenity" />
            <property key="name" />
        </featureType>

        <analysisSubject name="VgiAnalysisActionPerType" />
        <analysisSubject name="VgiAnalysisUserPerTags">
            <analysisSetting key="tagKey" value="" />
        </analysisSubject><!-- 
        <analysisSubject name="VgiAnalysisBatchGeneral" />
        <analysisSubject name="VgiAnalysisFeatureStability" />
        <analysisSubject name="VgiAnalysisTags">
            <analysisSetting key="tagKey" value="" />
        </analysisSubject>
        <analysisSubject name="VgiAnalysisOperationPerType" />
        <analysisSubject name="VgiAnalysisUserPerAction">
            <analysisSetting key="mergeActionTypes" value="false" />
        </analysisSubject>
        <analysisSubject name="VgiAnalysisBatchContributor" />
        <analysisSubject name="VgiAnalysisActionPerFeatureType" />
        <analysisSubject name="VgiAnalysisUserPerOperation">
            <analysisSetting key="mergeOperationTypes" value="false" />
        </analysisSubject>
        <analysisSubject name="VgiAnalysisActionDetails">
            <analysisSetting key="includeOperationDetails" value="true" />
        </analysisSubject>
        <analysisSubject name="VgiAnalysisHourOfDay" />-->
        <!-- <analysisSubject name="VgiAnalysisChangeDetection" />-->

        <actionDefinition name="AC_CREATE_POINT" actionType="CREATE" geometryType="POINT">
            <definitionRule operationType="OP_RECREATE_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_CREATE_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_TAG" entryPoint="IF_FEATURE_TYPE_ADDED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_ADDED" />
        </actionDefinition>
        <actionDefinition name="AC_CREATE_LINE" actionType="CREATE" geometryType="LINE">
            <definitionRule operationType="OP_RECREATE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_CREATE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_TAG" entryPoint="IF_FEATURE_TYPE_ADDED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_ADDED" />
            <definitionRule operationType="OP_ADD_NODE" entryPoint="NO" />
        </actionDefinition><!--
        <actionDefinition name="AC_CREATE_RELATION" actionType="CREATE" geometryType="RELATION">
            <definitionRule operationType="OP_RECREATE_RELATION" entryPoint="YES" />
            <definitionRule operationType="OP_CREATE_RELATION" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_TAG" entryPoint="IF_FEATURE_TYPE_ADDED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_ADDED" />
            <definitionRule operationType="OP_ADD_MEMBER" entryPoint="NO" />
        </actionDefinition>-->
        <actionDefinition name="AC_DELETE_POINT" actionType="DELETE" geometryType="POINT">
            <definitionRule operationType="OP_DELETE_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="IF_FEATURE_TYPE_REMOVED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_REMOVED" />
        </actionDefinition>
        <actionDefinition name="AC_DELETE_LINE" actionType="DELETE" geometryType="LINE">
            <definitionRule operationType="OP_DELETE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="IF_FEATURE_TYPE_REMOVED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_REMOVED" />
            <definitionRule operationType="OP_REMOVE_NODE" entryPoint="NO" />
        </actionDefinition><!--
        <actionDefinition name="AC_DELETE_RELATION" actionType="DELETE" geometryType="RELATION">
            <definitionRule operationType="OP_DELETE_RELATION" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="IF_FEATURE_TYPE_REMOVED" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="IF_FEATURE_TYPE_REMOVED" />
            <definitionRule operationType="OP_REMOVE_MEMBER" entryPoint="NO" />
        </actionDefinition>-->
        <actionDefinition name="AC_UPDATE_POINT" actionType="UPDATE" geometryType="POINT">
            <definitionRule operationType="OP_ADD_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_COORDINATE" entryPoint="YES" />
        </actionDefinition><!--
        <actionDefinition name="AC_SPLIT_MERGE_LINE" actionType="UPDATE" geometryType="LINE">
            <definitionRule operationType="OP_SPLIT_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_MERGE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_CREATE_WAY" entryPoint="NO" />
            <definitionRule operationType="OP_ADD_TAG" entryPoint="NO" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="NO" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="NO" />
            <definitionRule operationType="OP_REVERSE_WAY" entryPoint="NO" />
            <definitionRule operationType="OP_ADD_NODE" entryPoint="NO" />
            <definitionRule operationType="OP_REMOVE_NODE" entryPoint="NO" />
            <definitionRule operationType="OP_REORDER_NODE" entryPoint="NO" />
            <definitionRule operationType="OP_MODIFY_WAY_COORDINATE" entryPoint="NO" />
        </actionDefinition>-->
        <actionDefinition name="AC_UPDATE_LINE" actionType="UPDATE" geometryType="LINE">
            <definitionRule operationType="OP_SPLIT_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_MERGE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_REVERSE_WAY" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_REORDER_NODE" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_WAY_COORDINATE" entryPoint="YES" />
        </actionDefinition><!--
        <actionDefinition name="AC_UPDATE_RELATION" actionType="UPDATE" geometryType="RELATION">
            <definitionRule operationType="OP_ADD_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_TAG_VALUE" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_TAG" entryPoint="YES" />
            <definitionRule operationType="OP_ADD_MEMBER" entryPoint="YES" />
            <definitionRule operationType="OP_REMOVE_MEMBER" entryPoint="YES" />
            <definitionRule operationType="OP_REORDER_MEMBER" entryPoint="YES" />
            <definitionRule operationType="OP_MODIFY_ROLE" entryPoint="YES" />
        </actionDefinition>-->
    </analysis>
    <filterPolygons>
        <polygon label="Austria" geometry="POLYGON((15.161792 48.937221,15.243234 48.952930,15.266902 48.986675,15.357853 48.970810,15.681347 48.858466,15.818497 48.872315,15.833276 48.852006,15.885986 48.842033,15.875754 48.833093,15.899629 48.834746,15.931771 48.806428,16.083702 48.748719,16.358309 48.727259,16.384974 48.737078,16.435720 48.794542,16.519539 48.805601,16.643149 48.778264,16.675085 48.733926,16.798902 48.709224,16.873006 48.718991,16.896674 48.696977,16.910523 48.630780,16.945043 48.604166,16.946903 48.539726,16.864944 48.458077,16.844480 48.365602,16.901945 48.339402,16.898431 48.316173,16.916621 48.290800,16.954345 48.273127,16.981940 48.161299,17.047465 48.130991,17.080228 48.097608,17.070120 48.067665,17.075061 48.052081,17.069686 48.035674,17.148338 48.005443,17.085603 47.970148,17.095731 47.955730,17.075164 47.927773,17.113577 47.927127,17.067619 47.881626,17.004057 47.863281,17.055630 47.812354,17.040644 47.801114,17.050566 47.730989,17.075371 47.708484,16.902668 47.682026,16.864738 47.686729,16.850165 47.712929,16.797455 47.675463,16.730379 47.685902,16.689865 47.729568,16.567805 47.754192,16.531115 47.742978,16.512924 47.706004,16.431792 47.685463,16.407815 47.661330,16.630230 47.622004,16.656172 47.585934,16.650590 47.566555,16.689141 47.538030,16.677359 47.509866,16.636845 47.493200,16.640875 47.452919,16.481919 47.392302,16.456597 47.411836,16.433963 47.396850,16.424351 47.345199,16.473237 47.276805,16.421354 47.243138,16.412776 47.187173,16.441818 47.177121,16.433653 47.145754,16.497318 47.149681,16.509514 47.137537,16.454323 47.081701,16.493184 47.049145,16.424661 47.024082,16.467553 47.018423,16.486363 46.998554,16.415566 46.989433,16.274903 47.004315,16.223950 46.941064,16.110055 46.867916,15.981691 46.827685,15.970529 46.743014,16.014557 46.693714,16.016727 46.670641,15.986962 46.692190,15.850743 46.724488,15.755141 46.704024,15.652098 46.710819,15.652077 46.692760,15.545955 46.671881,15.511228 46.628369,15.462653 46.614649,15.388135 46.645578,15.061954 46.649557,15.004386 46.636844,14.967179 46.600257,14.933589 46.621135,14.850390 46.601136,14.788585 46.506646,14.709727 46.492512,14.679961 46.458871,14.590148 46.434428,14.540332 46.378643,14.502194 46.418356,14.450931 46.414481,14.395844 46.440991,14.149865 46.440061,14.038050 46.480963,13.860683 46.515250,13.782135 46.507782,13.585593 46.546927,13.478019 46.563567,13.231109 46.552173,13.146463 46.584961,12.830410 46.609637,12.739873 46.642994,12.547327 46.652192,12.405010 46.690123,12.370387 46.711155,12.342688 46.765131,12.269101 46.788566,12.266620 46.868148,12.126887 46.908869,12.141667 46.927989,12.111178 46.992998,12.203575 47.053331,12.204195 47.079686,12.180631 47.085215,12.014956 47.040489,11.899511 47.027725,11.734870 46.970622,11.683814 46.991913,11.596480 47.000440,11.521336 47.005821,11.485485 47.011250,11.411169 46.970493,11.313810 46.987262,11.174491 46.963853,11.091912 46.912435,11.052844 46.814921,10.996930 46.769110,10.870426 46.764072,10.834150 46.780246,10.722942 46.786448,10.716947 46.795207,10.748677 46.819443,10.662481 46.860965,10.486264 46.846366,10.453811 46.864427,10.458462 46.936619,10.378984 46.995505,10.313665 46.964318,10.295681 46.922693,10.235116 46.923313,10.201423 46.866830,10.111300 46.847116,9.875138 46.927421,9.862426 46.939772,9.870591 46.998838,9.857982 47.015478,9.669053 47.056199,9.581203 47.056870,9.615723 47.106764,9.581823 47.154901,9.552057 47.166890,9.562909 47.197740,9.535899 47.267428,9.593587 47.318589,9.613855 47.355595,9.663337 47.378032,9.664589 47.412651,9.661287 47.454465,9.579912 47.483483,9.547482 47.534547,9.676701 47.522630,9.738242 47.548064,9.767135 47.587277,9.807959 47.576322,9.813127 47.541750,9.945935 47.540769,9.982625 47.481031,10.022623 47.487955,10.080604 47.427390,10.053215 47.404860,10.082878 47.359074,10.209278 47.372484,10.190778 47.298251,10.159876 47.271121,10.239354 47.277735,10.305913 47.302178,10.371646 47.367394,10.428283 47.396049,10.451951 47.438759,10.447610 47.472581,10.419601 47.493045,10.430970 47.542164,10.414227 47.573015,10.429213 47.577020,10.457118 47.579242,10.453501 47.546246,10.482543 47.532862,10.536907 47.529865,10.583726 47.562473,10.655596 47.560455,10.760769 47.513638,10.858644 47.530666,10.892234 47.514879,10.851616 47.493097,10.858438 47.485165,10.959827 47.432506,10.955279 47.409614,10.979464 47.390545,11.083747 47.389512,11.193818 47.428941,11.215212 47.422998,11.213041 47.395816,11.237019 47.393956,11.305645 47.435400,11.383884 47.444857,11.388638 47.462220,11.365487 47.469326,11.412616 47.506094,11.551212 47.513690,11.589142 47.570379,11.620458 47.589654,11.820343 47.575288,11.851969 47.599163,11.935891 47.610738,12.212726 47.604020,12.205125 47.672285,12.176600 47.705823,12.242229 47.732023,12.223315 47.695901,12.238715 47.678900,12.424130 47.691560,12.444697 47.656291,12.496580 47.628903,12.653057 47.675127,12.761991 47.666859,12.751655 47.649392,12.813150 47.611824,12.785865 47.602677,12.778940 47.554825,12.931593 47.473512,13.001873 47.466019,13.037323 47.493045,13.028124 47.541647,13.038253 47.584022,13.072049 47.622417,13.072049 47.659469,13.019856 47.712929,12.964045 47.705358,12.892009 47.723548,12.945098 47.764294,12.941705 47.775383,12.991227 47.847106,12.931076 47.924879,12.862036 47.962552,12.830617 48.015468,12.750932 48.074741,12.736979 48.099959,12.745041 48.120630,12.862553 48.196646,12.933040 48.209255,13.032775 48.263567,13.274207 48.307104,13.415509 48.377157,13.459209 48.516394,13.440089 48.560965,13.454558 48.573445,13.520704 48.584581,13.624367 48.565357,13.716506 48.521691,13.733869 48.559776,13.774280 48.569207,13.802392 48.611711,13.800945 48.675221,13.816862 48.695582,13.783582 48.715270,13.815725 48.766430,13.982330 48.706485,14.034729 48.661165,14.006204 48.639513,14.015196 48.620393,14.074830 48.591712,14.325358 48.558639,14.405353 48.586286,14.458166 48.643182,14.533820 48.608714,14.594179 48.621323,14.605134 48.600316,14.634693 48.602332,14.675621 48.576235,14.695671 48.589542,14.703836 48.673103,14.775356 48.724004,14.800367 48.776507,14.939584 48.762813,14.978858 49.006260,15.137401 48.993031,15.141949 48.937479,15.161792 48.937221))" />
    </filterPolygons>
</settings>

Edit: now I'm recieving this error and message. Have you come across this? Is this normal?

21:00:22,422 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5720985/v5: Cannot find ref element POINT/3790937777 (2850000/2863325)
21:00:22,426 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5721374/v1: Cannot find ref element POINT/43140321 (2860000/2873328)
21:00:22,429 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5721776/v1: Cannot find ref element POINT/43158715 (2870000/2883328)
21:00:22,432 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5722202/v1: Cannot find ref element POINT/43175803 (2880000/2893328)
21:00:22,435 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5722610/v1: Cannot find ref element POINT/40818769 (2890000/2903434)
21:00:22,438 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5723081/v1: Cannot find ref element POINT/21915076 (2900000/2913434)
21:00:22,441 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5723652/v3: Cannot find ref element POINT/39376617 (2910000/2923434)
21:00:22,444 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5724218/v1: Cannot find ref element POINT/43239844 (2920000/2933434)
21:00:22,447 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5724731/v1: Cannot find ref element POINT/43253921 (2930000/2943434)
21:00:22,449 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5725026/v1: Cannot find ref element POINT/43268420 (2940000/2953434)
21:00:22,452 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5725369/v1: Cannot find ref element POINT/43282738 (2950000/2963524)
21:00:22,455 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5725745/v1: Cannot find ref element POINT/43296139 (2960000/2973628)
21:00:22,458 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5726085/v4: Cannot find ref element POINT/3423883388 (2970000/2983792)
21:00:22,461 ERROR [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  LINE/5726552/v1: Cannot find ref element POINT/43323786 (2980000/2993843)
21:00:22,463 INFO  [VgiOperationGeneratorDataHandlerImpl.addCoordinateToOperations()]  2986155 of 3000080 ref elements not found!
21:00:22,668 INFO  [VgiOperationGeneratorDataHandlerImpl.flushQueue()]  Polygon filter: 771 of 205009 features found!
21:00:22,668 INFO  [VgiOperationGeneratorDataHandlerImpl.flushQueue()]  Write 771 features to PBF files - Start
21:00:31,807 INFO  [VgiOperationGeneratorDataHandlerImpl.enqueueOperations()]  Sleep for 0.2 second...
21:00:32,008 INFO  [VgiOperationGeneratorDataHandlerImpl.enqueueOperations()]  #Parsed: 5706519674n|8184061w|0r #Features: 158610 - #ChildNodes: 3000054
21:00:32,008 INFO  [VgiOperationGeneratorDataHandlerImpl.flushQueue()]  Flush 158610 features with 3000054 child nodes
21:00:32,174 INFO  [VgiOperationGeneratorDataHandlerImpl.flushQueue()]  Find coordinates for 3000054 child nodes - Start
21:00:32,180 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,180 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,180 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,180 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,180 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,181 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 0 (of 6)
21:00:32,185 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,186 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,186 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,186 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,186 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,186 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 1 (of 6)
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,195 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 2 (of 6)
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,221 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 3 (of 6)
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,246 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 4 (of 6)
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]  Start reading PBF files
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]   - Data folder: /home/user/vgi-analytics-framework/pbf
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]   - Node ID Filter Size: 3000054
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]   - Way ID Filter Size: 0
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]   - Relation ID Filter Size: 0
21:00:32,280 INFO  [VgiOperationPbfReaderImpl.run()]   - producer: 5 (of 6)
SGroe commented 7 years ago

You can remove the word 'filterPolygon' and use the settings_op_generator.xml. This file only includes settings which are relevant for generating operations.

java -Xmx14g -jar vgi-analytics-framework-0.1.jar -importer -s ../settings/settings_op_generator.xml -o ../pbf/history-latest.osm.pbf

This error message appears if a way node (or relation member) is not found (e.g. outside of filter polygon). I have uploaded a new jar-file (vgi-analytics-framework-0.1.20170310.jar) which gives you a warning instead of an error in this case (and some other better error messages). Again, I appreciate feedback.

Import runtime: Global with tag filter for ski pistes and aerialways (as I did for a blog post): 4,5 hours Global: a few days Austria (from global history file with polygon filter): 10-12 hours You can also use the country-specific history files prepared by Geofabrik. If you just want to try the tool, select a small country (generating operations for Andorra takes a few seconds).

I would be happy if you help to write a usage documentation.

ktjaco commented 7 years ago

@SGroe Thank you for the information. But I don't see a CSV output. The CSV write outputs metadata.txt. Is this what you are referring to in the journal article?

12:17:57,875 INFO  [CSVFileWriter.close()]  File [/home/user/vgi-analytics-framework/results/2017-03-10T12-17-47Z/metadata.txt] written!

Also, what can be done further with the generated .pbf files? Would they have to be converted to .osm? My apologies regarding all of these questions.

SGroe commented 7 years ago

I can't tell you, why you don't see the CSV file...

You can use the PBF files to analyze the data. Just specify the PBF folder in the settings XML and analyze. The idea is that you generate the operations once and use these files for analyses.

ktjaco commented 7 years ago

@SGroe I just forgot to uncomment some of the settings in the XML files. Thank you again for this tool!

ktjaco commented 7 years ago

@SGroe Here is a look at some of the commands that I used. Let me know what you think, or if anything was missed.

https://github.com/ktjaco/vgi-analytics-framework/wiki/Documentation

SGroe commented 7 years ago

The documentation for setting up the tool on Ubuntu looks great. One remark: You don't need to set the -o flag for creating the quadtree and analysis. The OSM PBF file is only read during importing data.

SGroe commented 7 years ago

I have added a document giving short descriptions of the analysis methods. I will expand the document during the next days.