nsoft / jesterj

Document Ingestion Framework for Search Systems
Apache License 2.0
34 stars 33 forks source link

Emit visualization instead of just a dot #198

Closed nsoft closed 1 year ago

nsoft commented 1 year ago

As a stop gap before #192 is implemented, we can output dot notation visualization instead of a single "." every 5 seconds to help identify that the system is still running.

The result is like this

digraph "visualize" {
"Wikidocs_scanner" ["color"="blue","penwidth"="2.0","style"="filled","fillcolor"="white","label"="Wikidocs_scanner"]
"format_created_date" ["color"="black","penwidth"="2.0","style"="filled","fillcolor"="white","label"="0/1000\nformat_created_date"]
"format_modified_date" ["color"="black","penwidth"="2.0","style"="filled","fillcolor"="white","label"="0/1000\nformat_modified_date"]
"format_accessed_date" ["color"="black","penwidth"="2.0","style"="filled","fillcolor"="white","label"="0/1000\nformat_accessed_date"]
"size_to_int_step" ["color"="black","penwidth"="2.0","style"="filled","fillcolor"="white","label"="145/1000\nsize_to_int_step"]
"WikidocParser" ["color"="black","penwidth"="2.0","style"="filled","fillcolor"="0.7 0.3 1.0","label"="1000/1000\nWikidocParser\n(RoundRobinToSolr)"]
"solr_sender" ["color"="red","penwidth"="2.0","style"="filled","fillcolor"="white","label"="1000/1000\nsolr_sender"]
"Wikidocs_scanner" -> "format_created_date"
"format_created_date" -> "format_modified_date"
"format_modified_date" -> "format_accessed_date"
"format_accessed_date" -> "size_to_int_step"
"size_to_int_step" -> "WikidocParser"
"WikidocParser" -> "solr_sender"
}

which can be rendered like this:

image

This is so super nifty and fairly low risk that I'm going to let it creep into 1.0

nsoft commented 1 year ago

The 145/1000 for size_to_int_step shows that 145 documents are waiting in queue at that step for processing out of 1000 possible (the step batch size)

epugh commented 1 year ago

Communication of what is going on is super valuable!