Pickle Error with MongoTrials

[jengelman]

Running hyperopt on Windows 7, regular trials works fine, but when I try to serialize into MongoDB, I get the following error:

Traceback (most recent call last):
  File "C:\Users\jengelman\Desktop\svn\silo\users\jengelman\utilities\optimizer.py", line 186, in <module>
    trials = optimize(d0, d1, models, space, extra_params)
  File "C:\Users\jengelman\Desktop\svn\silo\users\jengelman\utilities\optimizer.py", line 149, in optimize
    trials=trials
  File "C:\Anaconda3\lib\site-packages\hyperopt\fmin.py", line 307, in fmin
    return_argmin=return_argmin,
  File "C:\Anaconda3\lib\site-packages\hyperopt\base.py", line 635, in fmin
    return_argmin=return_argmin)
  File "C:\Anaconda3\lib\site-packages\hyperopt\fmin.py", line 318, in fmin
    verbose=verbose)
  File "C:\Anaconda3\lib\site-packages\hyperopt\fmin.py", line 77, in __init__
    domain, protocol=self.pickle_protocol)
TypeError: can't pickle SwigPyObject objects

Tried switching pickle protocols to dill, but ran into the same error. (I also had to comment out lines 1183 and 1185 to get this far, since windows doesn't recognize SIGHUP or SIGUSR1).

Has anyone managed to get MongoTrials working on Windows/have a workaround?

[ivan-marroquin]

Hi Jengelman,

I don't have a workaround to share with you. I am also experiencing issues to run Mongo Trials on a Windows 7 machine (see my ticket #388).

Where do you get stuck? Perhaps, if we join efforts ... we could find a way to overcome this issue.

Ivan

[jve2kor]

Did you have any solution for the problem yet?

[ivan-marroquin]

I couldn't find a way to make it work on a Windows machine. I am hoping that the team behind hyperopt will come up with a solution.

[jengelman]

I ended up writing a custom version of Trials by directly connecting and storing results in Mongo during each evaluation, instead of using the version given in hyperopt. If we wanted to pickle SwigPyObjects, we'd have to define__getstate__ and __setstate__ in the relevant object so the serializers will work, I'll take a look when I have some more time.

[ivan-marroquin]

Thanks jengelman for looking into this. Does your implementation work on windows? Can I get a copy of it? I would like to give it a try.

[maxpumperla]

@jengelman I'd be very interested in a pointer to what you're doing. Can you create a gist? doesn't have to be polished.

[github-actions[bot]]

This issue has been marked as stale because it has been open 120 days with no activity. Remove the stale label or comment or this will be closed in 30 days.