Better data ingest - Githubissues

DaveParr / mtgai

Recommendation tool for the 99 in your commander deck

MIT License

2 stars 1 forks source link

Better data ingest #2

Closed DaveParr closed 5 months ago

DaveParr commented 5 months ago

Make

Download cards data via make
Start app with make
Create vectorstore with make
migrate chroma data ingest to make
Refactor make data into separate data commands

Structure

Write cards to parquet
Remove gradio app
Restructure data layout to beter use data science cookie cutter

Data processung

Utilise sets data from mtgjson in sets filtering
Filter by cutoff_date and type of set
Pass date to card filter as cutoff_date

Documentation

Document .env
Document data processing locations

Bug

fix bug where procesed_data.name was the set not the card name

Logging

simplify and unify logging to struct log