organisciak
commented
8 years ago Seems unnecessarily complex for a very specific use. I've been training based on seeded hashes, can something like that be implemented for random but predictable ordering? With your approach, would you check for id collisions, or just have the id sufficiently large that it can be reasonable expected not to happen?
On Fri, Jun 24, 2016, 9:20 AM Benjamin Schmidt notifications@github.com wrote:
A niche case for some extensions is to use a bookworm word count index as an input to a batched machine learning process. I sometimes want to train a neural network on word counts, for example.
A SELECT * FROM master_bookcounts query will provide this data, sorted by bookid. Most large text corpora have some pre-specified order. For example, Chronicling America is arrange by newspaper, and then by year within each newspaper. This will make learning outputs less effective, since each batch should be as representative of the whole as possible. A gradient descent algorithm might lurch quite far off in the wrong direction, or even (gasp) find and settle into a local maximum.
If bookids are randomly assigned, rather than assigned sequentially, there's a workaround for this: the primary table keys will allow SELECT * FROM master_bookcounts ORDER BY bookid to be delivered nearly as fast as a table scan, but it will now have a random order.
Since bookids are arbitrary and internal, I don't see a downside here. It's possible, though, that index construction might take longer if the tables aren't sorted to begin with. So that's something to watch out for.
This might create problems for additions of bookids down the road. Therefore, the random ids should be drawn from the full range 1,16777215, which is the maximum bookid supported under the standard scheme, not just the range 1,catalog_length.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bookworm-project/BookwormDB/issues/103, or mute the thread https://github.com/notifications/unsubscribe/AADj721eVnVYAPR-lEzLUk7xjVS7VXoFks5qO_WzgaJpZM4I93Ym .
bmschmidt
A niche case for some extensions is to use a bookworm word count index as an input to a batched machine learning process. I sometimes want to train a neural network on word counts, for example.
A
SELECT * FROM master_bookcountsquery will provide this data, sorted by bookid. Most large text corpora have some pre-specified order. For example, Chronicling America is arrange by newspaper, and then by year within each newspaper. This will make learning outputs less effective, since each batch should be as representative of the whole as possible. A gradient descent algorithm might lurch quite far off in the wrong direction, or even (gasp) find and settle into a local maximum.If bookids are randomly assigned, rather than assigned sequentially, there's a workaround for this: the primary table keys will allow
SELECT * FROM master_bookcounts ORDER BY bookidto be delivered nearly as fast as a table scan, but it will now have a random order.Since bookids are arbitrary and internal, I don't see a downside here. It's possible, though, that index construction might take longer if the tables aren't sorted to begin with. So that's something to watch out for.
This might create problems for additions of bookids down the road. Therefore, the random ids should be drawn from the full range
1,16777215, which is the maximum bookid supported under the standard scheme, not just the range1,catalog_length.