High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.
Reading from disk is by far the slowest part of the training for us as the parsing is single threaded and blocking. This change reduced our entire end to end training time by 4 times and it is even slightly faster than reading from memory as that involves one run of single threaded reading from disk.
This stores upto 10 next parsed blocks in memory and could have hardware_concurrency number of blocks in processing. So the upper bound of memory is fixed and is independent of size of data.
Reading from disk is by far the slowest part of the training for us as the parsing is single threaded and blocking. This change reduced our entire end to end training time by 4 times and it is even slightly faster than reading from memory as that involves one run of single threaded reading from disk.
This stores upto 10 next parsed blocks in memory and could have
hardware_concurrencynumber of blocks in processing. So the upper bound of memory is fixed and is independent of size of data.