Text support - Githubissues

code-lukas / Spacewalker

MIT License

4 stars 0 forks source link

Text support #1

Open code-lukas opened 5 months ago

code-lukas commented 5 months ago

Add text support to spacewalker:

Steps: Convention for input:

csv
one column has to be called 'Text'
one column has to be 'Id'

Convention for Triton / Onnx:

Tokenizer on triton? Or inside the python inference function? -> Check best practices
HuggingFace support (?) -> Get models as ONNX directly from HuggingFace

Visualisation:

How to store text in Minio as to break as little as possible
Preview in the annotation GUI (as image or as text)

code-lukas commented 5 months ago

Tokenizer should be inside triton to allow simple queries later.