Dynamic-prompting

[JasonLo]

18 and #14

Updates to the text-to-graph pipeline:

• Dynamic Prompting: Prior to initiating the graph extraction process, the system will now scan the input text to identify any known entities. If such entities are found, their names will be included in the prompt to guide the model in prioritizing these entities during extraction.
• Entity Alignment Post-Processing: Occasionally, the LLM may generate an entity similar to a known entity. We will use semantic embedding techniques to align these closely related entities with their corresponding known entities.

[JasonLo]

Significant updates have been made that could impact your current branch, @ilmcconnell. I'll set up the API endpoint tomorrow, so you won't need to run it on your local machine to test the endpoint.

[JasonLo]

Deployed on COSMOS2: http://cosmos0002.chtc.wisc.edu:4510/llm_grpah

[ilmcconnell]

@JasonLo nice work. LGTM.

I'm curious why you use relative imports, rather than explicit imports? i.e. from .macrostrat import ... vs from text2graph.macrostrat import ...

Curious to see some performance numbers. I wonder if those embeddings might work even better with some sentence context?

[JasonLo]

@JasonLo nice work. LGTM.

I'm curious why you use relative imports, rather than explicit imports? i.e. from .macrostrat import ... vs from text2graph.macrostrat import ...

Curious to see some performance numbers. I wonder if those embeddings might work even better with some sentence context?

For relative import, it is just for convenient.

Regarding embedding for post-processing alignment, achieving precise performance metrics may be challenging without a labeled dataset. When considering sentence context for entity matching, it's unclear how effective this will be. Macrostrat, the source entity, lacks sentence context, which may lead to complications or unexpected outcomes when attempting to use sentence context with newly discovered entities. However, if you believe exploring sentence context could be beneficial, we can delve into this idea further.