TypeError("cannot serialize '_io.BufferedReader' object")

[varundey]

~/Downloads/himawaripy-master > himawaripy
Updating...
Latest version: 2016/07/31 19:10:00 GMT
Offset version: 2016/07/31 14:28:00 GMT

Downloading tiles: 0/16 completedTraceback (most recent call last):
  File "/usr/local/bin/himawaripy", line 9, in <module>
    load_entry_point('himawaripy==1.1', 'console_scripts', 'himawaripy')()
  File "/usr/local/lib/python3.5/dist-packages/himawaripy-1.1-py3.5.egg/himawaripy/himawaripy.py", line 88, in main
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f435fe7ebe0>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

[boramalper]

Which Python version are you using?

[varundey]

I installed the package using python 3: sudo python3 setup.py install as given over here. It produces the above error on running himawaripy in the fourth step.

[boramalper]

Honestly speaking, I don't know what might have caused the error, but it's probably when the downloader tries to return tiledata (here).

The thing is, we are not trying to return _io.BufferedReader object at all. Is it possible that by any chance you removed the parentheses of the function call at line 54 in himawaripy.py? It should be like this:

tiledata = tile_w.read()

Lastly, can you please tell which Python version exactly you are using? Like 3.5.2

Sorry for the inconvenience!

[varundey]

Hi @boramalper , python3 --version returns Python 3.5.2 and no I did not modify the source in any way :smile: Hopefully, there will be a solution to this :+1: :grinning:

[boramalper]

@ZSandyGhost wrote:

Updating...
Latest version: 2016/08/05 16:50:00 GMT
Offset version: 2016/08/05 12:08:00 GMT

Downloading tiles: 0/64 completedTraceback (most recent call last):
  File "/usr/local/bin/himawaripy", line 9, in <module>
    load_entry_point('himawaripy==1.1', 'console_scripts', 'himawaripy')()
  File "/usr/local/lib/python3.5/dist-packages/himawaripy-1.1-py3.5.egg/himawaripy/himawaripy.py", line 88, in main
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f9eb1756c50>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

I get this error when I run it from terminal. This is the first run. I got this error on Ubuntu few weeks ago when I first tried it. Now I'm on Mint 18 and getting the same error.

[ZSandyGhost]

I'm also using Python 3.5.2

[ZSandyGhost]

I edited the download_chunk function in himawaripy.py file to something like this so it pritnts out the urls for individual parts of the image and then I went to those urls manually from the browser and it gave me 404 Not found. Could this be the problem? Maybe it blocks certain ip addresses or something. I tried using an VPN and still no luck

url = url_format.format(level, width, strftime("%Y/%m/%d/%H%M%S", latest), x, y)
print(url)

[ZSandyGhost]

I found the issue that causing this. It's the timezones as mentioned in #68 I'm from asia and It worked after I changed my timezone to New York and it worked like a charm

And setting auto_offset to False in config.py also have solved the problem without altering the timezones

[varundey]

@ZSandyGhost is right. It's the issue in timezone. Though changing auto_offset is not necessary. I changed the timezone to Asia/Kolkata and it works like a charm. I think the timezone has to be behind Japanese time-zone. Not sure though, just a thought!

[xgt001]

setting auto_offset = False worked for me as well. Thanks guys

[collif]

I have been receiving this error as well, however changing auto_offset to False did not work for me. Any other potential work arounds?

[boramalper]

Hello all,

still being not sure, I suspect this was because of multiprocessing (Python was probably failing to serialize and pass huge objects back and forth different processes). Now that we switched to multi-threading, the issue should be resolved, hopefully.

Can you please make a clean installation of v2 (https://github.com/boramalper/himawaripy/tree/v2) and let me know if it fixes your problem?

Thanks!

[smenon8]

I am trying to download images that I scraped from Flickr. And I ran into the same issue.

63 [21:00:03] [sreejithmenon@Sreejiths-MacBook-Pro:script] $ python SocialMediaImageExtracts.py
Traceback (most recent call last):
  File "SocialMediaImageExtracts.py", line 83, in <module>
    multiProcDownload(download_dir, urlList)
  File "SocialMediaImageExtracts.py", line 50, in multiProcDownload
    r.get()
  File "/Users/sreejithmenon/anaconda/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x1037ac940>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

I am using the below code to download the files:

def multiProcDownload(download_dir, urlList):
    start_time = time.time()

    download = partial(download_link, download_dir)
    with Pool(10) as p:
        r = p.map(download, urlList)

    print("Time elapsed: %f" %(time.time() - start_time))

Will a multithreading approach work?

[boramalper]

@smenon8 How is this relevant to himawaripy?