As most rnaseq and other data files will be compressed, we should be using a library properly suited for streaming large compressed files. The smart_open library is made for this, and can also stream from AWS S3, Hadoop and straight HTTP servers which is pretty useful.
As most rnaseq and other data files will be compressed, we should be using a library properly suited for streaming large compressed files. The smart_open library is made for this, and can also stream from AWS S3, Hadoop and straight HTTP servers which is pretty useful.