search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

hyperband-optimization notebook demo fails to serialize #118

Closed murphyk closed 1 year ago

murphyk commented 3 years ago

I am running this demo: https://cloud.coiled.io/examples/jobs/hyperband-optimization. It fails on this line

search.fit(X_train, y_train2)

I re-ran it with a smaller hyper-parameter search space (see below), and just using data frame for 1 month of data (not full year), but it still fails.

Here is the reduced search space


#params = {
#    "module__activation": ["relu", "elu", "softsign", "leaky_relu", "rrelu"],
 #   "batch_size": [32, 64, 128, 256],
#    "optimizer__lr": loguniform(1e-4, 1e-3),
#    "optimizer__weight_decay": loguniform(1e-6, 1e-3),
#    "optimizer__momentum": uniform(0, 1),
#    "optimizer__nesterov": [True],
#}

params = {
    "module__activation": ["relu", "elu", ],
    "batch_size": [32, 64],
    "optimizer__lr": loguniform(1e-4, 1e-3),
    "optimizer__weight_decay": loguniform(1e-6, 1e-3),
    "optimizer__momentum": uniform(0, 1),
    "optimizer__nesterov": [True],
}

And here is the output

[CV, bracket=2] creating 9 models
[CV, bracket=1] creating 5 models
[CV, bracket=0] creating 3 models
[CV, bracket=0] For training there are between 155034 and 165984 examples in each chunk

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 229, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
distributed.comm.utils - INFO - Unserializable Message: [{'op': 'update-graph-hlg', 'hlg': [b'', b"\x81\xa6layers\x91\x83\xaa__module__\xc0\xa8__name__\xc0\xa5state\x83\xa3dsk\x89\xd91_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x04\x80\x05\x95\xf9\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c._create_model-edfd857639b04dc3fd094488a1ae7b69\x94\x8c7('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\x94\x8c7('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd91_partial_fit-7a4ab64e-b428-4972-bd3b-6158073a7739\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x05\x80\x05\x95\xfa\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c1_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\x94\x8c6('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\x94\x8c6('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd91_partial_fit-75f48cda-2d1f-47cc-aed4-dcfe2b8351c8\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x05\x80\x05\x95\xfa\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c1_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\x94\x8c6('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\x94\x8c6('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd9+_score-bfb3d8eb-3207-490d-9c89-c45a57d124c1\x82\xa8function\xc4>\x80\x04\x953\x00\x00\x00\x00\x00\x00\x00\x8c$dask_ml.model_selection._incremental\x94\x8c\x06_score\x94\x93\x94.\xa4args\xc4\xcd\x80\x04\x95\xc2\x00\x00\x00\x00\x00\x00\x00(\x8c1_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\x94\x8c)finalize-49a7de52a40a751230747cac92f32da7\x94\x8c)finalize-226d5e6d88116370058ad6c45f615bd0\x94\x8c\x17sklearn.metrics._scorer\x94\x8c\x13_passthrough_scorer\x94\x93\x94t\x94.\xd91_partial_fit-c23b8d8d-04ce-420c-8263-738d803c389f\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x05\x80\x05\x95\xfa\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c1_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\x94\x8c6('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\x94\x8c6('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd91_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x04\x80\x05\x95\xf9\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c._create_model-f4ce1ef425cf269fb7f457bee38119ee\x94\x8c7('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\x94\x8c7('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd9+_score-551f2f84-34aa-4d7f-a834-fe4f435f85c7\x82\xa8function\xc4>\x80\x04\x953\x00\x00\x00\x00\x00\x00\x00\x8c$dask_ml.model_selection._incremental\x94\x8c\x06_score\x94\x93\x94.\xa4args\xc4\xcd\x80\x04\x95\xc2\x00\x00\x00\x00\x00\x00\x00(\x8c1_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\x94\x8c)finalize-49a7de52a40a751230747cac92f32da7\x94\x8c)finalize-226d5e6d88116370058ad6c45f615bd0\x94\x8c\x17sklearn.metrics._scorer\x94\x8c\x13_passthrough_scorer\x94\x93\x94t\x94.\xd91_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\x82\xae__Serialized__\xc3\xa4data\x92\x81\xaaserializer\xa6pickle\x91\xc5\x01\x04\x80\x05\x95\xf9\x00\x00\x00\x00\x00\x00\x00(\x8c$dask_ml.model_selection._incremental\x94\x8c\x0c_partial_fit\x94\x93\x94\x8c._create_model-3fcc7fea795fc070d123aa6494a5bfe1\x94\x8c7('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\x94\x8c7('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\x94\x8c\x08builtins\x94\x8c\x04dict\x94\x93\x94]\x94\x86\x94t\x94.\xd9+_score-7d7217bc-392f-4874-bd55-779d491c591c\x82\xa8function\xc4>\x80\x04\x953\x00\x00\x00\x00\x00\x00\x00\x8c$dask_ml.model_selection._incremental\x94\x8c\x06_score\x94\x93\x94.\xa4args\xc4\xcd\x80\x04\x95\xc2\x00\x00\x00\x00\x00\x00\x00(\x8c1_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\x94\x8c)finalize-49a7de52a40a751230747cac92f32da7\x94\x8c)finalize-226d5e6d88116370058ad6c45f615bd0\x94\x8c\x17sklearn.metrics._scorer\x94\x8c\x13_passthrough_scorer\x94\x93\x94t\x94.\xacdependencies\x89\xd91_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\x93\xd97('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\xd97('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\xd9._create_model-edfd857639b04dc3fd094488a1ae7b69\xd91_partial_fit-7a4ab64e-b428-4972-bd3b-6158073a7739\x93\xd96('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\xd91_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\xd96('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\xd9+_score-7d7217bc-392f-4874-bd55-779d491c591c\x93\xd9)finalize-49a7de52a40a751230747cac92f32da7\xd91_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\xd9)finalize-226d5e6d88116370058ad6c45f615bd0\xd91_partial_fit-75f48cda-2d1f-47cc-aed4-dcfe2b8351c8\x93\xd96('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\xd91_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\xd96('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\xd9+_score-bfb3d8eb-3207-490d-9c89-c45a57d124c1\x93\xd9)finalize-49a7de52a40a751230747cac92f32da7\xd91_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\xd9)finalize-226d5e6d88116370058ad6c45f615bd0\xd91_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b\x93\xd9._create_model-f4ce1ef425cf269fb7f457bee38119ee\xd97('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\xd97('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\xd9+_score-551f2f84-34aa-4d7f-a834-fe4f435f85c7\x93\xd9)finalize-49a7de52a40a751230747cac92f32da7\xd91_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\xd9)finalize-226d5e6d88116370058ad6c45f615bd0\xd91_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8\x93\xd9._create_model-3fcc7fea795fc070d123aa6494a5bfe1\xd97('concatenate-6955afb52c51819c024cf2467264c941', 27, 0)\xd97('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 27, 0)\xd91_partial_fit-c23b8d8d-04ce-420c-8263-738d803c389f\x93\xd96('concatenate-6955afb52c51819c024cf2467264c941', 9, 0)\xd91_partial_fit-767432bd-806f-4970-bc61-be7b13349a67\xd96('concatenate-0d5f01c3db36e8c196adf0e999a3dcd6', 9, 0)\xabannotations\xc0"], 'keys': ['_partial_fit-767432bd-806f-4970-bc61-be7b13349a67', '_partial_fit-7a4ab64e-b428-4972-bd3b-6158073a7739', '_partial_fit-75f48cda-2d1f-47cc-aed4-dcfe2b8351c8', '_score-bfb3d8eb-3207-490d-9c89-c45a57d124c1', '_partial_fit-c23b8d8d-04ce-420c-8263-738d803c389f', '_partial_fit-07e00992-cfd5-4e3c-a95a-a03f97ff999b', '_score-551f2f84-34aa-4d7f-a834-fe4f435f85c7', '_partial_fit-db796f45-c0ae-4a36-b42d-4215f3155df8', '_score-7d7217bc-392f-4874-bd55-779d491c591c'], 'priority': None, 'submitting_task': None, 'fifo_timeout': '60s', 'annotations': {'priority': {(Delayed('_partial_fit-c23b8d8d-04ce-420c-8263-738d803c389f'), Delayed('_partial_fit-7a4ab64e-b428-4972-bd3b-6158073a7739'), Delayed('_partial_fit-75f48cda-2d1f-47cc-aed4-dcfe2b8351c8')): -1}}, 'actors': None}, {'op': 'client-releases-keys', 'keys': ['len-358ff13e5c612bf42102743cbb87cc8f'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-3c783f28dce04bcee6f3dbecfadc6b0e'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-1e18463b091628f1296300c7a8268a86'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-cae1e666ea0312f68df5ffdaaaf55408'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-b928af5ecb23321de020029066ce60a0'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-d70511943b845f9b09165297425792f0'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-23f101a628f001b4a86d1fbe50cff8fb'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-516c7916ca9eff9f45185b79c42af311'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-e496ce34a6bbd1c4eefbb78b66fb91ae'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-7f8ffdd5eae9a83d77a5617182c4f19b'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-61fb83545bd81644492c88f2e82a740a'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-30dbd8572d6749f739f644a56b63927c'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-d7da5b3d01a31a9a21897fd290aed0cf'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-2940b713115673e30e6f0f8106fe9559'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-aeed0fe54262d090782d034d18c32c7a'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-4351f1a7e9719137b59972764f58ff08'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-eea526db7812cd8c60e33515d98b5cca'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-da42f447bda86fa9183fa19c8bda1f9e'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-b1743124b094a9434f9b68ab21bfc476'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-c2282d533f0aa0d55cb57feffc0bb3b2'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-c8627e962c4879682ee35b6fa5fa5ed4'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-2159a9ea99a91d103bafc9171cfccbb2'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-6f57c9e19ccd8d192c5db6f18d67e66a'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-90378d90da6b5ee7484607aacddf3a08'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-ac66584a570df69ca38d8c805547bc85'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-844c6138fa424c04f934c0b70c1142ae'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-b3d2613c64326f5ad88066c66f6a9371'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-a9e96f2590a91c8e7eade559f5f10a67'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-a75670066c60a0136649bbb44c5c6ac9'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-6dc5e61bad85e8ebc224f2a1fe3ff27c'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-5a4fc3b852be4b85b6abb0ad21f22bde'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-ad084b594b69ee676da38a5432704cfc'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-f3ba88eab25683f89b635b7cb957e1a3'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-a281af7c3687859266e1d36ff83440c2'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-5dfc8371dea366919b37098e09ae7da5'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-00b813a1301cfeecd13a6f750b3cb945'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-cc28ce7fe2847a047a0a73e7a3c6ca44'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-b998fb8401444cb4ac90912d33c2d702'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-46525ad3c9adf81aee465a3d2f5856dc'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-3cfe8b4791560373194734a996064e6d'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}, {'op': 'client-releases-keys', 'keys': ['len-69503024016f5c9ca875f4e2e7386e97'], 'client': 'Client-618ec31c-7862-11eb-805d-0a58a9feac02'}]
distributed.comm.utils - ERROR - can not serialize 'Delayed' object
Traceback (most recent call last):
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/comm/utils.py", line 32, in _to_frames
    protocol.dumps(
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 229, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/batched.py", line 93, in _background_send
    nbytes = yield self.comm.write(
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/comm/tcp.py", line 230, in write
    frames = await to_frames(
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/comm/utils.py", line 52, in to_frames
    return _to_frames()
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/comm/utils.py", line 32, in _to_frames
    protocol.dumps(
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/core.py", line 184, in dumps_msgpack
    payload = msgpack.dumps(msg, default=msgpack_encode_default, use_bin_type=True)
  File "/opt/conda/envs/coiled/lib/python3.8/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 229, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
---------------------------------------------------------------------------
CancelledError                            Traceback (most recent call last)
<ipython-input-13-438707e8dfcb> in <module>
----> 1 search.fit(X_train, y_train2)

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask_ml/model_selection/_incremental.py in fit(self, X, y, **fit_params)
    704         client = default_client()
    705         if not client.asynchronous:
--> 706             return client.sync(self._fit, X, y, **fit_params)
    707         return self._fit(X, y, **fit_params)
    708 

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    836             return future
    837         else:
--> 838             return sync(
    839                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    840             )

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    338     if error[0]:
    339         typ, exc, tb = error[0]
--> 340         raise exc.with_traceback(tb)
    341     else:
    342         return result[0]

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/utils.py in f()
    322             if callback_timeout is not None:
    323                 future = asyncio.wait_for(future, callback_timeout)
--> 324             result[0] = yield future
    325         except Exception as exc:
    326             error[0] = sys.exc_info()

/opt/conda/envs/coiled/lib/python3.8/site-packages/tornado/gen.py in run(self)
    760 
    761                     try:
--> 762                         value = future.result()
    763                     except Exception:
    764                         exc_info = sys.exc_info()

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask_ml/model_selection/_hyperband.py in _fit(self, X, y, **fit_params)
    399         _brackets_ids = list(reversed(sorted(SHAs)))
    400 
--> 401         _SHAs = await asyncio.gather(
    402             *[SHAs[b]._fit(X, y, **fit_params) for b in _brackets_ids]
    403         )

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask_ml/model_selection/_incremental.py in _fit(self, X, y, **fit_params)
    650 
    651         with context:
--> 652             results = await fit(
    653                 self.estimator,
    654                 self._get_params(),

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask_ml/model_selection/_incremental.py in fit(model, params, X_train, y_train, X_test, y_test, additional_calls, fit_params, scorer, random_state, verbose, prefix)
    464         A history of all models scores over time
    465     """
--> 466     return await _fit(
    467         model,
    468         params,

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask_ml/model_selection/_incremental.py in _fit(model, params, X_train, y_train, X_test, y_test, additional_calls, fit_params, scorer, random_state, verbose, prefix)
    188         y_test = y_test.to_dask_array()
    189 
--> 190     X_train, y_train, X_test, y_test = dask.persist(X_train, y_train, X_test, y_test)
    191 
    192     if isinstance(X_test, da.Array):

/opt/conda/envs/coiled/lib/python3.8/site-packages/dask/base.py in persist(*args, **kwargs)
    753             else:
    754                 if client.get == schedule:
--> 755                     results = client.persist(
    756                         collections, optimize_graph=optimize_graph, **kwargs
    757                     )

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/client.py in persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
   2942         names = {k for c in collections for k in flatten(c.__dask_keys__())}
   2943 
-> 2944         futures = self._graph_to_futures(
   2945             dsk,
   2946             names,

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/client.py in _graph_to_futures(self, dsk, keys, workers, allow_other_workers, priority, user_priority, resources, retries, fifo_timeout, actors)
   2541                 dsk = HighLevelGraph.from_collections(id(dsk), dsk, dependencies=())
   2542 
-> 2543             dsk = highlevelgraph_pack(dsk, self, keyset)
   2544 
   2545             annotations = {}

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/highlevelgraph.py in highlevelgraph_pack(hlg, client, client_keys)
    113                 "__module__": None,
    114                 "__name__": None,
--> 115                 "state": _materialized_layer_pack(
    116                     layer,
    117                     hlg.get_all_external_keys(),

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/protocol/highlevelgraph.py in _materialized_layer_pack(layer, all_keys, known_key_dependencies, client, client_keys)
     42             )
     43         if stringify(future.key) not in client.futures:
---> 44             raise CancelledError(stringify(future.key))
     45     unpacked_futures_deps = {}
     46     for k, v in dsk.items():

CancelledError: ('reshape-266960d81106894da56ae23936ddf831', 16, 0
murphyk commented 3 years ago

FWIW, my local laptop has these versions

distributed.version '2021.02.0' dask.version '2021.02.0' coiled.version '0.0.36'

murphyk commented 3 years ago

Interestingly, I get exactly the same error when I run the script version of the notebook on my laptop, as opposed to running jupyter notebook on coiled.

CV, bracket=2] creating 9 models
[CV, bracket=1] creating 5 models
[CV, bracket=0] creating 3 models
[CV, bracket=0] For training there are between 155034 and 165984 examples in each chunk
distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/Users/kpmurphy/opt/anaconda3/lib/python3.8/site-packages/distributed/protocol/core.py", line 39, in dumps
    small_header, small_payload = dumps_msgpack(msg, **compress_opts)
shughes-uk commented 1 year ago

This is quite stale and we no longer maintain these specific examples

ncclementi commented 1 year ago

If this is referencing to the hyper parameter optimization notebook, this is an updated version that works https://docs.coiled.io/user_guide/examples/dask-optuna-hpo.html