BingAds / BingAds-Python-SDK

Other
117 stars 162 forks source link

Memory usage #66

Closed pocin closed 7 years ago

pocin commented 7 years ago

I am having some memory issues when downloading reports. I checked a large portion of the source code and it seems that iterators and streaming is used all over the place. However this line bogs me - https://github.com/BingAds/BingAds-Python-SDK/blob/67c00e9e14b6565c5f1e6f3f3c2fbd6cabd3f5b0/bingads/reporting/reporting_operation.py#L128 doesn't it result in reading the whole file in memory?

and shouldn't it be something along these lines?

                with contextlib.closing(zipfile.ZipFile(zip_file_path)) as compressed:
                    first = compressed.namelist()[0]
                    with open(result_file_path, 'wb') as f, compressed.open(first, 'r') as c:
                        for line in c:
                            f.write(c)
pocin commented 7 years ago

I actually made some quick and dirty benchmarks and this would be best to use:

import shutil
                with contextlib.closing(zipfile.ZipFile(zip_file_path)) as compressed:
                    first = compressed.namelist()[0]
                    with open(result_file_path, 'wb') as f, compressed.open(first, 'r') as c:
                        shutil.copyfileobj(c, f)

this consumes almost zero memory

eric-urban commented 7 years ago

@pocin we updated bulk_operation and reporting_operation.