Open eshellman opened 3 months ago
I looked a little at the csv metadata and it seems it does not contain some info like publisher, rights, num downloads. I am not sure that all books have 100% utf8 files with uniform headers but where they do I think I already use them in the textget function if I remember correctly, they still need some minor cleanup if people plan to use them in ml otherwise they will bias their data with the uniform header fields. Thanks for the info though, I will need to investigate more, as I would glaly ditch rdf. .
The source txt files that gutenbergpy uses do indeed need cleanup and are not uniformly UTF-8, which is why we now generate clean utf8 text files.
Neither the RDF files nor the CSV files contain original publisher information, we're working on it, but we'll need help to make it reasably complete.
Rights info is contained in the uniform header.
Let me know if I can answer any questions!
Eric
On Jul 28, 2024, at 7:27 AM, raduangelescu @.***> wrote:
I looked a little at the csv metadata and it seems it does not contain some info like publisher, rights, num downloads. I am not sure that all books have 100% utf8 files with uniform headers but where they do I think I already use them in the textget function if I remember correctly, they still need some minor cleanup if people plan to use them in ml otherwise they will bias their data with the uniform header fields. Thanks for the info though, I will need to investigate more, as I would glaly ditch rdf. .
— Reply to this email directly, view it on GitHub https://github.com/raduangelescu/gutenbergpy/issues/23#issuecomment-2254349151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHCGMO34YDTAUF2F6Q5BATZOR6NZAVCNFSM6AAAAABLSALRWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJUGM2DSMJVGE. You are receiving this because you authored the thread.
Much of the code in this repo here is no longer needed because PG now supplies clean text files, 100% utf8, with uniform headers at urls like https://gutenberg.org/cache/epub/65869/pg65869.txt Also available are csv metadata files and a zipped tar archive of the txt files at https://gutenberg.org/cache/epub/feeds/ (these are regularly updated.)