Inference against a custom dataset

artificialwisdomai / origin

Artificial Wisdom™ Cloud Platform

Apache License 2.0

3 stars 4 forks source link

Inference against a custom dataset #93

Open rstarmer opened 1 year ago

rstarmer commented 1 year ago

If possible extend PyKoi (or a similar toolset) to inference against any/all of the following datasets:

ArXiv
pubmed
wikipedia

sdake commented 11 months ago

@rstarmer do you permit changes in the input format? We generally have far more elaborate user stories. I guess we didn't do that here. Currently, for storage of retrieval information, jsonl is an easy path forward. Let me see if I can enhance this user story a bit.

Please also note, that processing realnews took quite a bit of time. I have found the wikipedia jsonl files, and it will take about (roughly guessing) the same compute....