Keras Integration Issues

[irfannurafif]

@janvanrijn Let's discuss Keras integration problem here.

As said by Joaquin:

We have a similar issue with Keras models, which have the entire network expressed as a hyperparameter. This is a working example: https://www.openml.org/r/9201688 but if the network gets too big we run into the same issue.

[janvanrijn]

I see that you try to serialize the build fn as a json. This seems bad practice, as the following two pieces of json are considered the same, whereas OpenML will not identify them as such:

{"layer0": [["class_name", "Reshape"], ["config", {"batch_input_shape": [null, 784], "dtype": "float32", "name": "reshape_19", "target_shape": [28, 28, 1], "trainable": true}]], "layer1": [["class_name", "Conv2D"], ["config", {"activation": "relu", "activity_regularizer": null, "batch_input_shape": [null, 28, 28, 1], "bias_constraint": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "bias_regularizer": null, "data_format": "channels_last", "dilation_rate": [1, 1], "dtype": "float32", "filters": 20, "kernel_constraint": null, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "mode": "fan_avg", "scale": 1.0, "seed": null}}, "kernel_regularizer": null, "kernel_size": [5, 5], "name": "conv2d_37", "padding": "same", "strides": [1, 1], "trainable": true, "use_bias": true}]]}

{"layer1": [["class_name", "Conv2D"], ["config", {"activation": "relu", "activity_regularizer": null, "batch_input_shape": [null, 28, 28, 1], "bias_constraint": null, "bias_initializer": {"class_name": "Zeros", "config": {}}, "bias_regularizer": null, "data_format": "channels_last", "dilation_rate": [1, 1], "dtype": "float32", "filters": 20, "kernel_constraint": null, "kernel_initializer": {"class_name": "VarianceScaling", "config": {"distribution": "uniform", "mode": "fan_avg", "scale": 1.0, "seed": null}}, "kernel_regularizer": null, "kernel_size": [5, 5], "name": "conv2d_37", "padding": "same", "strides": [1, 1], "trainable": true, "use_bias": true}]], "layer0": [["class_name", "Reshape"], ["config", {"batch_input_shape": [null, 784], "dtype": "float32", "name": "reshape_19", "target_shape": [28, 28, 1], "trainable": true}]]}

Wouldn't it be possible to break down this build_fn in logical pieces? e.g., each layer is stored in a separate hyperparameter. Of course, this makes serializing a little bit harder as a "9-layer" NN is now a different flow than a "8-layer" NN, but I think this is the cleaner solution.

Is this possible? Or are there also other things stored in the build_fn besides layers? Can you know beforehand which?

What do you think?

[irfannurafif]

Wouldn't it be possible to break down this build_fn in logical pieces? e.g., each layer is stored in a separate hyperparameter.

By break-downing into separate hyperparameter means that the flow will contains additional row fields for each layer, to describe the configuration in the associated layer, doesn't it? Maybe that could work. Will try to implement it and report the result.

Or are there also other things stored in the build_fn besides layers? Can you know beforehand which?

build_fn only storing the string function to build Keras model. Actually my approach of storing model configuration into build_fn is not good practice afterall since it does not has correct values. Will try implementing above solution and restoring the build_fn value to default value for serializing function parameter (eg: {"oml-python:serialized_object": "function", "value": "main.create_model"} )

[janvanrijn]

Btw, it would probably be good to not upload the results to the main server, but to https://test.openml.org/ (for now)

[joaquinvanschoren]

Hmm, this means that all neural nets with 8 layers are the same flow, and all neural nets with 9 layers are another? That seems incongruous (and confusing)?

Would it make sense to define a flow by the layer types, so you would get a flow named keras.Reshape.Conv2D.MaxPooling2D.Conv2D.MaxPooling2D and then you specify the hyperparameters in each layer via hyperparameters named consistently 'layer_0', 'layer_1' or maybe 'build_fn__layer0' etc (notice the double underscore).

That's a lot more recognizable and structured?

That would also be more consistent with the way we represent pipelines.

[janvanrijn]

Hmm, this means that all neural nets with 8 layers are the same flow, and all neural nets with 9 layers are another?

Yes.

That seems incongruous (and confusing)?

That is highly subjective.

Would it make sense to define a flow by the layer types, so you would get a flow named keras.Reshape.Conv2D.MaxPooling2D.Conv2D.MaxPooling2D and then you specify the hyperparameters in each layer via hyperparameters named consistently 'layer_0', 'layer_1' or maybe 'build_fn__layer0' etc (notice the double underscore).

How is that different from my suggestion? (You still have different flows for networks of different sizes)

[joaquinvanschoren]

If you had the same idea, I misunderstood, and we agree :).

What I thought you meant is that all keras networks with 8 layers would have the same general name, all have 8 hyperparameters 'layer_0' ... 'layer_7' and only differ in the value of these hyperparameters.

A naming scheme as I stated above could have many networks of length 8 with different flow names. So, you would have different flows for networks of different sizes and for different structures. A network keras.Reshape.Conv2D.Conv2D would be a different flow than keras.Reshape.Conv2D.MaxPooling2D.

[janvanrijn]

If you had the same idea

I'm not a Neural Network guy, so I wouldn't claim that

I misunderstood, and we agree :).

Great.

A naming scheme as I stated above could have many networks of length 8 with different flow names. So, you would have different flows for networks of different sizes and for different structures. A network keras.Reshape.Conv2D.Conv2D would be a different flow than keras.Reshape.Conv2D.MaxPooling2D.

sounds like another problem solved. @irfannurafif what are your thoughts?

[irfannurafif]

A naming scheme as I stated above could have many networks of length 8 with different flow names. So, you would have different flows for networks of different sizes and for different structures. A network keras.Reshape.Conv2D.Conv2D would be a different flow than keras.Reshape.Conv2D.MaxPooling2D.

This could works as short term solution for describing small network. As for bigger and complex network, I guess the same problem in the beginning (maximum field length) will persist for the flow name. But I think for the current constraint (2048 chars IIRC), maybe its more than enough for now.

[joaquinvanschoren]

I think this will work for the foreseeable future. If need be, we can find ways to shorten the name based on what makes more sense, and even use specific names for well-known deep learning architectures, like keras.AlexNet :)

[janvanrijn]

Although I would like to prevent it, making the flow name (a bit) longer is actually less of an issue

[irfannurafif]

Here is my latest run http://www.openml.org/r/9201743 . Is it already as good as the suggestion above?

I don't know how to test the run on the test server, so I just upload it on the main server.

[joaquinvanschoren]

That looks good (and I'll need to some design work to shorten the hyperparameter names on screen anyways).

@Jan: do we have a style guide for flow names (e.g. do we allow parentheses, commas, etc?)

On Fri, 25 May 2018 at 12:30 Irfan Nur Afif notifications@github.com wrote:

Here is my latest run http://www.openml.org/r/9201743 . Is it as suggested?

I don't know how to test the run on the test server, so I just upload it on the main server.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/openml/OpenML/issues/751#issuecomment-392012081, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQVzBZYC-OzYTe4Ct056fnVDhsEhUtks5t191IgaJpZM4UMfsA .

-- Thank you, Joaquin

[janvanrijn]

Can you please use the test server to upload rums while the package is in development?

The URL is https://test.openml.org/

[joaquinvanschoren]

To use the test server, change the url in the configuration file (~/.openml/config)

On Fri, 25 May 2018 at 14:13 janvanrijn notifications@github.com wrote:

Can you please use the test server to upload rums while the package is in development?

The URL is https://test.openml.org/

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/openml/OpenML/issues/751#issuecomment-392035298, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV8V6aCQJttFPAR7i-7Ejb6aq03NYks5t1_V7gaJpZM4UMfsA .

-- Thank you, Joaquin

[jan]

Who summoned me?

[joaquinvanschoren]

Sorry, @jan, I used the wrong github handle. Should have been @janvanrijn

[irfannurafif]

Done. Reuploaded on test server https://test.openml.org/r/10381