Enhance loader features and architecture

scottkleinman / lexos

Development repo for the Lexos API

MIT License

1 stars 0 forks source link

Enhance loader features and architecture #4

Closed scottkleinman closed 1 year ago

scottkleinman commented 2 years ago

Potential new loader features:

Read pdf, docx, jsonl, and csv formats
Remove Project Gutenberg boilerplate

It's also worth considering whether the loader should be implemented through Python dataclasses or pydantic BaseModel.

Since the current loader is implemented in the io.basic module, it would be easy to develop these in a separate module and perhaps merge them later on.

scottkleinman commented 2 years ago

Progress report:

docx and pdf loading have been added to the new smart loader. Work on json(l) and csv formats is in progress.
Removing Project Gutenberg boilerplate is more appropriate for the Scrubber module. A function has been added to the remove component, and I'll push it as soon as I've written the documentation.

scottkleinman commented 1 year ago

Progress report:

json(l) and csv formats have been added through the dataset.
There is no tutorial description of the process of removing Project Gutenberg headers, but the API is fully documented.

The remaining item in this issue is the general enhancement to the loader architecture. However, the issue was initially based on the older basic module, rather than the more recent smart and dataset modules. So I'm going to close that and revisit enhancements to those modules separately.