search

Oslandia / osm-data-classification

Migrated to: https://gitlab.com/Oslandia/osm-data-classification
MIT License
24 stars 6 forks source link

Failed dependencies #1

Closed ktjaco closed 6 years ago

ktjaco commented 7 years ago

Hello,

Thank you so much for this tool!

I've been testing it out and I've come across some failed tasks. I've installed the required dependencies but I still get an output with tasks failed or left pending. Here is a screenshot of my console.

Any help is much appreciated!

user@ubuntu:~/osGeo/osm-data-classification$ PYTHONPATH=src python3 -m luigi --local-scheduler --module analysis_tasks AutoKMeans --dsname prince-edward-island

/home/user/.local/lib/python3.5/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools
2017-10-12 17:11:32,227 :: INFO :: instance : Loaded []
DEBUG: Checking if AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,279 :: DEBUG :: check_complete : Checking if AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,281 :: DEBUG :: check_complete : Checking if KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,282 :: DEBUG :: check_complete : Checking if KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
INFO: Informed scheduler that task   AutoKMeans_data_prince_edward_is_user_17626ff787   has status   PENDING
2017-10-12 17:11:32,283 :: INFO :: _add_task : Informed scheduler that task   AutoKMeans_data_prince_edward_is_user_17626ff787   has status   PENDING
INFO: Informed scheduler that task   KMeansAnalysis_data_prince_edward_is_user_17626ff787   has status   PENDING
2017-10-12 17:11:32,283 :: INFO :: _add_task : Informed scheduler that task   KMeansAnalysis_data_prince_edward_is_user_17626ff787   has status   PENDING
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2) is complete
2017-10-12 17:11:32,286 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3) is complete
2017-10-12 17:11:32,287 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4) is complete
2017-10-12 17:11:32,287 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5) is complete
2017-10-12 17:11:32,288 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6) is complete
2017-10-12 17:11:32,288 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9) is complete
INFO: Informed scheduler that task   KMeansReport_data_prince_edward_is_user_17626ff787   has status   PENDING
2017-10-12 17:11:32,290 :: INFO :: _add_task : Informed scheduler that task   KMeansReport_data_prince_edward_is_user_17626ff787   has status   PENDING
DEBUG: Checking if AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2017-10-12 17:11:32,292 :: DEBUG :: check_complete : Checking if AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
DEBUG: Checking if PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2017-10-12 17:11:32,293 :: DEBUG :: check_complete : Checking if PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
DEBUG: Checking if PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2017-10-12 17:11:32,293 :: DEBUG :: check_complete : Checking if PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd   has status   PENDING
2017-10-12 17:11:32,294 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd   has status   PENDING
INFO: Informed scheduler that task   PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5   has status   PENDING
2017-10-12 17:11:32,295 :: INFO :: _add_task : Informed scheduler that task   PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5   has status   PENDING
DEBUG: Checking if VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
2017-10-12 17:11:32,296 :: DEBUG :: check_complete : Checking if VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
INFO: Informed scheduler that task   PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5   has status   PENDING
2017-10-12 17:11:32,297 :: INFO :: _add_task : Informed scheduler that task   PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5   has status   PENDING
DEBUG: Checking if MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user) is complete
2017-10-12 17:11:32,298 :: DEBUG :: check_complete : Checking if MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user) is complete
INFO: Informed scheduler that task   VarianceAnalysisTask_data_prince_edward_is__de6ea788c3   has status   PENDING
2017-10-12 17:11:32,298 :: INFO :: _add_task : Informed scheduler that task   VarianceAnalysisTask_data_prince_edward_is__de6ea788c3   has status   PENDING
DEBUG: Checking if OSMElementEnrichment(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,299 :: DEBUG :: check_complete : Checking if OSMElementEnrichment(datarep=data, dsname=prince-edward-island) is complete
DEBUG: Checking if AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5) is complete
2017-10-12 17:11:32,300 :: DEBUG :: check_complete : Checking if AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5) is complete
INFO: Informed scheduler that task   MetadataNormalization_data_prince_edward_is_user_37dbbaf168   has status   PENDING
2017-10-12 17:11:32,301 :: INFO :: _add_task : Informed scheduler that task   MetadataNormalization_data_prince_edward_is_user_37dbbaf168   has status   PENDING
DEBUG: Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
2017-10-12 17:11:32,302 :: DEBUG :: check_complete : Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
DEBUG: Checking if UserMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,302 :: DEBUG :: check_complete : Checking if UserMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task   AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60   has status   PENDING
2017-10-12 17:11:32,303 :: INFO :: _add_task : Informed scheduler that task   AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60   has status   PENDING
DEBUG: Checking if ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,304 :: DEBUG :: check_complete : Checking if ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task   UserMetadataExtract_data_prince_edward_is_ee0c385c15   has status   PENDING
2017-10-12 17:11:32,304 :: INFO :: _add_task : Informed scheduler that task   UserMetadataExtract_data_prince_edward_is_ee0c385c15   has status   PENDING
INFO: Informed scheduler that task   ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15   has status   PENDING
2017-10-12 17:11:32,305 :: INFO :: _add_task : Informed scheduler that task   ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15   has status   PENDING
INFO: Informed scheduler that task   EditorCountByUser_data_5_2262508398   has status   DONE
2017-10-12 17:11:32,305 :: INFO :: _add_task : Informed scheduler that task   EditorCountByUser_data_5_2262508398   has status   DONE
DEBUG: Checking if OSMHistoryParsing(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,306 :: DEBUG :: check_complete : Checking if OSMHistoryParsing(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task   OSMElementEnrichment_data_prince_edward_is_ee0c385c15   has status   PENDING
2017-10-12 17:11:32,306 :: INFO :: _add_task : Informed scheduler that task   OSMElementEnrichment_data_prince_edward_is_ee0c385c15   has status   PENDING
INFO: Informed scheduler that task   OSMHistoryParsing_data_prince_edward_is_ee0c385c15   has status   PENDING
2017-10-12 17:11:32,307 :: INFO :: _add_task : Informed scheduler that task   OSMHistoryParsing_data_prince_edward_is_ee0c385c15   has status   PENDING
DEBUG: Checking if PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2017-10-12 17:11:32,308 :: DEBUG :: check_complete : Checking if PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
INFO: Informed scheduler that task   AutoPCA_data_prince_edward_is__dba8e840cb   has status   PENDING
2017-10-12 17:11:32,308 :: INFO :: _add_task : Informed scheduler that task   AutoPCA_data_prince_edward_is__dba8e840cb   has status   PENDING
INFO: Informed scheduler that task   PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb   has status   PENDING
2017-10-12 17:11:32,309 :: INFO :: _add_task : Informed scheduler that task   PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_d62355b835   has status   PENDING
2017-10-12 17:11:32,310 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_d62355b835   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f   has status   PENDING
2017-10-12 17:11:32,311 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_107c39fe48   has status   PENDING
2017-10-12 17:11:32,312 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_107c39fe48   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_23ac327dcf   has status   PENDING
2017-10-12 17:11:32,313 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_23ac327dcf   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_3d109ea54e   has status   PENDING
2017-10-12 17:11:32,314 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_3d109ea54e   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_2afd311993   has status   PENDING
2017-10-12 17:11:32,315 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_2afd311993   has status   PENDING
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd   has status   PENDING
2017-10-12 17:11:32,317 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd   has status   PENDING
INFO: Done scheduling tasks
2017-10-12 17:11:32,317 :: INFO :: _schedule_and_run : Done scheduling tasks
INFO: Running Worker with 1 processes
2017-10-12 17:11:32,317 :: INFO :: run : Running Worker with 1 processes
DEBUG: Asking scheduler for work...
2017-10-12 17:11:32,317 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:11:32,317 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:11:32,317 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 22
2017-10-12 17:11:32,318 :: DEBUG :: run : Pending tasks: 22
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
2017-10-12 17:11:32,318 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
<TRACE> Initialization of a TimelineHandler instance !
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
2017-10-12 17:12:46,953 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:12:46,954 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   OSMHistoryParsing_data_prince_edward_is_ee0c385c15   has status   DONE
2017-10-12 17:12:46,954 :: INFO :: _add_task : Informed scheduler that task   OSMHistoryParsing_data_prince_edward_is_ee0c385c15   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:12:46,954 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:12:46,954 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:12:46,955 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 21
2017-10-12 17:12:46,955 :: DEBUG :: run : Pending tasks: 21
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
2017-10-12 17:12:46,955 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
/home/user/.local/lib/python3.5/site-packages/numpy/lib/arraysetops.py:463: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask |= (ar1 == a)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
2017-10-12 17:13:32,723 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:13:32,723 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   OSMElementEnrichment_data_prince_edward_is_ee0c385c15   has status   DONE
2017-10-12 17:13:32,724 :: INFO :: _add_task : Informed scheduler that task   OSMElementEnrichment_data_prince_edward_is_ee0c385c15   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:13:32,724 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:13:32,724 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:13:32,724 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 20
2017-10-12 17:13:32,725 :: DEBUG :: run : Pending tasks: 20
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:13:32,725 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:06,386 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:06,386 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15   has status   DONE
2017-10-12 17:14:06,387 :: INFO :: _add_task : Informed scheduler that task   ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:06,387 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:06,387 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:06,387 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 19
2017-10-12 17:14:06,388 :: DEBUG :: run : Pending tasks: 19
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   UserMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:06,388 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   UserMetadataExtract(datarep=data, dsname=prince-edward-island)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      UserMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:14,528 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      UserMetadataExtract(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:14,529 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   UserMetadataExtract_data_prince_edward_is_ee0c385c15   has status   DONE
2017-10-12 17:14:14,529 :: INFO :: _add_task : Informed scheduler that task   UserMetadataExtract_data_prince_edward_is_ee0c385c15   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:14,529 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:14,529 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:14,530 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 18
2017-10-12 17:14:14,530 :: DEBUG :: run : Pending tasks: 18
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
2017-10-12 17:14:14,530 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
2017-10-12 17:14:15,679 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:15,680 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60   has status   DONE
2017-10-12 17:14:15,680 :: INFO :: _add_task : Informed scheduler that task   AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:15,681 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:15,681 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:15,681 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 17
2017-10-12 17:14:15,681 :: DEBUG :: run : Pending tasks: 17
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
2017-10-12 17:14:15,681 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
2017-10-12 17:14:22,565 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:22,566 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   MetadataNormalization_data_prince_edward_is_user_37dbbaf168   has status   DONE
2017-10-12 17:14:22,566 :: INFO :: _add_task : Informed scheduler that task   MetadataNormalization_data_prince_edward_is_user_37dbbaf168   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:22,566 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:22,567 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:22,567 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 16
2017-10-12 17:14:22,567 :: DEBUG :: run : Pending tasks: 16
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
2017-10-12 17:14:22,567 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
2017-10-12 17:14:22,589 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:22,589 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   VarianceAnalysisTask_data_prince_edward_is__de6ea788c3   has status   DONE
2017-10-12 17:14:22,590 :: INFO :: _add_task : Informed scheduler that task   VarianceAnalysisTask_data_prince_edward_is__de6ea788c3   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:22,590 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:22,590 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:22,590 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 15
2017-10-12 17:14:22,591 :: DEBUG :: run : Pending tasks: 15
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:22,591 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,022 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,023 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb   has status   DONE
2017-10-12 17:14:23,023 :: INFO :: _add_task : Informed scheduler that task   PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,023 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,023 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,023 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 14
2017-10-12 17:14:23,024 :: DEBUG :: run : Pending tasks: 14
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,024 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,182 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,183 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   AutoPCA_data_prince_edward_is__dba8e840cb   has status   DONE
2017-10-12 17:14:23,183 :: INFO :: _add_task : Informed scheduler that task   AutoPCA_data_prince_edward_is__dba8e840cb   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,183 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,183 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,183 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 13
2017-10-12 17:14:23,184 :: DEBUG :: run : Pending tasks: 13
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,184 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,906 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,907 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5   has status   DONE
2017-10-12 17:14:23,907 :: INFO :: _add_task : Informed scheduler that task   PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,908 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,908 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,908 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 12
2017-10-12 17:14:23,908 :: DEBUG :: run : Pending tasks: 12
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,908 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:24,874 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:24,875 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5   has status   DONE
2017-10-12 17:14:24,876 :: INFO :: _add_task : Informed scheduler that task   PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:24,876 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:24,876 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:24,876 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 11
2017-10-12 17:14:24,876 :: DEBUG :: run : Pending tasks: 11
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
2017-10-12 17:14:24,876 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
2017-10-12 17:14:25,304 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:25,304 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_23ac327dcf   has status   DONE
2017-10-12 17:14:25,305 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_23ac327dcf   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:25,305 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:25,305 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:25,305 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 10
2017-10-12 17:14:25,306 :: DEBUG :: run : Pending tasks: 10
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
2017-10-12 17:14:25,306 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
2017-10-12 17:14:25,764 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:25,764 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_107c39fe48   has status   DONE
2017-10-12 17:14:25,765 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_107c39fe48   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:25,765 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:25,765 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:25,765 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 9
2017-10-12 17:14:25,766 :: DEBUG :: run : Pending tasks: 9
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
2017-10-12 17:14:25,766 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
2017-10-12 17:14:26,272 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:26,272 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f   has status   DONE
2017-10-12 17:14:26,273 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:26,273 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:26,273 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:26,273 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 8
2017-10-12 17:14:26,274 :: DEBUG :: run : Pending tasks: 8
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
2017-10-12 17:14:26,274 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
2017-10-12 17:14:26,834 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:26,835 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_d62355b835   has status   DONE
2017-10-12 17:14:26,835 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_d62355b835   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:26,835 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:26,836 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:26,836 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 7
2017-10-12 17:14:26,836 :: DEBUG :: run : Pending tasks: 7
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
2017-10-12 17:14:26,836 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
2017-10-12 17:14:27,182 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,182 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_2afd311993   has status   DONE
2017-10-12 17:14:27,183 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_2afd311993   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,183 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,183 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,183 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 6
2017-10-12 17:14:27,184 :: DEBUG :: run : Pending tasks: 6
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
2017-10-12 17:14:27,184 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
2017-10-12 17:14:27,579 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,580 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_3d109ea54e   has status   DONE
2017-10-12 17:14:27,581 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_3d109ea54e   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,581 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,581 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,581 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 5
2017-10-12 17:14:27,581 :: DEBUG :: run : Pending tasks: 5
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
2017-10-12 17:14:27,581 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
2017-10-12 17:14:27,779 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,779 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd   has status   DONE
2017-10-12 17:14:27,780 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,780 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,780 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,780 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 4
2017-10-12 17:14:27,781 :: DEBUG :: run : Pending tasks: 4
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
2017-10-12 17:14:27,781 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
2017-10-12 17:14:28,377 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,377 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd   has status   DONE
2017-10-12 17:14:28,378 :: INFO :: _add_task : Informed scheduler that task   KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,378 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,378 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,378 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 3
2017-10-12 17:14:28,378 :: DEBUG :: run : Pending tasks: 3
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,379 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,490 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done      KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,491 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansReport_data_prince_edward_is_user_17626ff787   has status   DONE
2017-10-12 17:14:28,491 :: INFO :: _add_task : Informed scheduler that task   KMeansReport_data_prince_edward_is_user_17626ff787   has status   DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,492 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,492 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,492 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 2
2017-10-12 17:14:28,493 :: DEBUG :: run : Pending tasks: 2
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,493 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running   KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
ERROR: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) failed    KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 194, in run
    new_deps = self._run_get_new_deps()
  File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 131, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/user/osGeo/osm-data-classification/src/analysis_tasks.py", line 910, in run
    self.nbmin_clusters, self.nbmax_clusters)
  File "/home/user/osGeo/osm-data-classification/src/unsupervised_learning.py", line 451, in kmeans_elbow_silhouette
    s = random.sample(range(len(feature)), 2000)
  File "/usr/lib/python3.5/random.py", line 315, in sample
    raise ValueError("Sample larger than population")
ValueError: Sample larger than population
2017-10-12 17:14:28,608 :: ERROR :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) failed    KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 194, in run
    new_deps = self._run_get_new_deps()
  File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 131, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/user/osGeo/osm-data-classification/src/analysis_tasks.py", line 910, in run
    self.nbmin_clusters, self.nbmax_clusters)
  File "/home/user/osGeo/osm-data-classification/src/unsupervised_learning.py", line 451, in kmeans_elbow_silhouette
    s = random.sample(range(len(feature)), 2000)
  File "/usr/lib/python3.5/random.py", line 315, in sample
    raise ValueError("Sample larger than population")
ValueError: Sample larger than population
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,632 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   KMeansAnalysis_data_prince_edward_is_user_17626ff787   has status   FAILED
2017-10-12 17:14:28,639 :: INFO :: _add_task : Informed scheduler that task   KMeansAnalysis_data_prince_edward_is_user_17626ff787   has status   FAILED
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,639 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,639 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,639 :: INFO :: prune : Done pruning task graph
DEBUG: Done
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : Done
DEBUG: There are no more tasks to run at this time
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) was stopped. Shutting down Keep-Alive thread
2017-10-12 17:14:28,641 :: INFO :: run : Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) was stopped. Shutting down Keep-Alive thread
INFO: 
**===== Luigi Execution Summary =====

Scheduled 23 tasks of which:
* 1 present dependencies were encountered:
    - 1 EditorCountByUser(datarep=data, n_top_editor=5)
* 20 ran successfully:
    - 1 AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
    - 1 AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
    - 1 ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
    - 8 KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2...9)
    - 1 KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
    ...
* 1 failed:
    - 1 KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

2017-10-12 17:14:28,648 :: INFO :: _schedule_and_run : 
===== Luigi Execution Summary =====

Scheduled 23 tasks of which:
* 1 present dependencies were encountered:
    - 1 EditorCountByUser(datarep=data, n_top_editor=5)
* 20 ran successfully:
    - 1 AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
    - 1 AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
    - 1 ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
    - 8 KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2...9)
    - 1 KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
    ...
* 1 failed:
    - 1 KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====
delhomer commented 6 years ago

Hi @ktjaco!

Thank you for your nice comment!

By checking your log, it seems that the problem does not come from a missing dependency. How many users are there in your dataset? I bet it is less than 2000. If so, the problem comes from the way we are computing the cluster silhouette (as it is computer-intensive, we sample only 2000 users and do the computation on them, however if you have less user than this threshold, the 'sample' function will crash). We have fixed this problem, this should work fine now. Do not hesitate to open a new issue if another problem arise!

ktjaco commented 6 years ago

@delhomer I was able to follow the blog posts and generate some statistics on tagsets anyway. Thank you again for this framework!