[RoadMap] - TensorLayer 2.0

[DEKHTIARJonathan]

NETWORK API REFACTORING - TO DO LIST

Finished work:

- **activation.py:** * PReluLayer: - [x] refactored - [x] tested * PRelu6Layer: - [x] refactored - [x] tested * PTRelu6Layer: - [x] refactored - [x] tested - **convolution/** * AtrousConv1dLayer: - [ ] refactored => Need to be implemented - [ ] tested * AtrousConv2dLayer: - [x] refactored - [x] tested * AtrousDeConv2dLayer: - [x] refactored - [x] tested * BinaryConv2d: - [x] refactored - [x] tested * Conv1d: - [x] refactored - [x] tested * Conv2d: - [x] refactored - [x] tested * Conv1dLayer: - [x] refactored - [x] tested * Conv2dLayer: - [x] refactored - [x] tested * Conv3dLayer: - [x] refactored - [x] tested * DeConv2d: - [x] refactored - [x] tested * DeConv3d: - [x] refactored - [x] tested * DeConv2dLayer: - [x] refactored - [x] tested * DeConv3dLayer: - [x] refactored - [x] tested * DeformableConv2d: - [x] refactored - [x] tested * DepthwiseConv2d: - [x] refactored - [x] tested * DorefaConv2d: - [x] refactored - [x] tested * GroupConv2d: - [x] refactored - [x] tested * QuantizedConv2d: - [x] refactored - [x] tested * QuantizedConv2dWithBN: - [x] refactored - [x] tested * SeparableConv1d: - [x] refactored - [x] tested * SeparableConv2d: - [x] refactored - [x] tested * SubpixelConv1d: - [x] refactored - [x] tested * SubpixelConv2d: - [x] refactored - [x] tested * TernaryConv2d: - [x] refactored - [x] tested - **dense/** * BinaryDenseLayer: - [x] refactored - [x] tested * DenseLayer: - [x] refactored - [x] tested * DorefaDenseLayer: - [x] refactored - [x] tested * DropconnectDenseLayer: - [x] refactored - [x] tested * QuantizedDense: - [x] refactored - [x] tested * QuantizedDenseWithBN: - [x] refactored - [x] tested * TernaryDenseLayer: - [x] refactored - [x] tested - **dropout.py** * DropoutLayer: - [x] refactored - [x] tested - **extend.py** * ExpandDimsLayer: - [x] refactored - [x] tested * TileLayer: - [x] refactored - [x] tested - **image_resampling.py** * UpSampling2dLayer: - [x] refactored - [x] tested * DownSampling2dLayer: - [x] refactored - [x] tested - **importer.py** * SlimNetsLayer: - [x] refactored - [x] tested * KerasLayer: - [x] refactored - [x] tested - **inputs.py** * InputLayer: - [x] refactored - [x] tested * OneHotInputLayer: - [x] refactored - [x] tested * Word2vecEmbeddingInputlayer: - [x] refactored - [x] tested * EmbeddingInputlayer: - [x] refactored - [x] tested * AverageEmbeddingInputlayer: - [x] refactored - [x] tested - **lambda_layers.py** * ElementwiseLambdaLayer: - [x] refactored - [x] tested * LambdaLayer: - [x] refactored - [x] tested - **merge.py** * ConcatLayer: - [x] refactored - [x] tested * ElementwiseLayer: - [x] refactored - [x] tested - **noise.py** * GaussianNoiseLayer: - [x] refactored - [x] tested - **normalization.py** * BatchNormLayer: - [x] refactored - [x] tested * GroupNormLayer: - [x] refactored - [x] tested * InstanceNormLayer: - [x] refactored - [x] tested * LayerNormLayer: - [x] refactored - [x] tested * LocalResponseNormLayer: - [x] refactored - [x] tested * SwitchNormLayer: - [x] refactored - [x] tested - **padding.py** * PadLayer: - [x] refactored - [x] tested * ZeroPad1d: - [x] refactored - [x] tested * ZeroPad2d: - [x] refactored - [x] tested * ZeroPad3d: - [x] refactored - [x] tested - **pooling/** * MaxPool1d: - [x] refactored - [x] tested * MaxPool2d: - [x] refactored - [x] tested * MaxPool3d: - [x] refactored - [x] tested * MeanPool1d: - [x] refactored - [x] tested * MeanPool2d: - [x] refactored - [x] tested * MeanPool3d: - [x] refactored - [x] tested * GlobalMaxPool1d: - [x] refactored - [x] tested * GlobalMaxPool2d: - [x] refactored - [x] tested * GlobalMaxPool3d: - [x] refactored - [x] tested * GlobalMeanPool1d: - [x] refactored - [x] tested * GlobalMeanPool2d: - [x] refactored - [x] tested * GlobalMeanPool3d: - [x] refactored - [x] tested * PoolLayer: - [x] refactored - [x] tested - **quantize_layers.py** * SignLayer: - [x] refactored - [x] tested - **recurrent/** * BiDynamicRNNLayer: - [x] refactored - [x] tested * BiRNNLayer: - [x] refactored - [x] tested * ConvLSTMLayer: - [x] refactored - [x] tested * DynamicRNNLayer: - [x] refactored - [x] tested * RNNLayer: - [x] refactored - [x] tested * Seq2Seq: - [x] refactored - [x] tested - **reshape.py** * FlattenLayer: - [x] refactored - [x] tested * ReshapeLayer: - [x] refactored - [x] tested * TransposeLayer: - [x] refactored - [x] tested - **scale.py** * ScaleLayer: - [x] refactored - [x] tested

Unittests Status:

• test_activations.py
  • [ ] refactored
  • [x] tested
• test_array_ops.py
  • [x] refactored
  • [x] tested
• test_decorators.py
  • [x] refactored
  • [x] tested
• test_documentation.py
  • [x] refactored
  • [x] tested
• test_layers_activation.py
  • [x] refactored
  • [x] tested
• test_layers_basic.py
  • [x] refactored
  • [ ] tested
• test_layers_convolution_1d.py
  • [x] refactored
  • [ ] tested
• test_layers_convolution_2d.py
  • [x] refactored
  • [ ] tested
• test_layers_convolution_3d.py
  • [x] refactored
  • [ ] tested
• test_layers_convolution_deformable.py
  • [x] refactored
  • [ ] tested
• test_layers_core.py
  • [x] refactored
  • [ ] tested
• test_layers_extend.py
  • [x] refactored
  • [ ] tested
• test_layers_flow_control.py removed in favour of eager mode
• test_layers_importer.py
  • [x] refactored
  • [ ] tested
• test_layers_merge.py
  • [x] refactored
  • [ ] tested
• test_layers_normalization.py
  • [x] refactored
  • [ ] tested
• test_layers_padding.py
  • [x] refactored
  • [ ] tested
• test_layers_pooling.py
  • [x] refactored
  • [ ] tested
• test_layers_recurrent.py
  • [x] refactored
  • [ ] tested
• test_layers_reshape.py
  • [x] refactored
  • [ ] tested
• test_layers_spatial_transformer.py
  • [x] refactored
  • [ ] tested
• test_layers_stack.py
  • [x] refactored
  • [ ] tested
• test_layers_super_resolution.py
  • [x] refactored
  • [ ] tested
• test_layers_time_distributed.py
  • [x] refactored
  • [ ] tested
• test_logging.py
  • [x] refactored
  • [x] tested
• test_logging_hyperdash.py
  • [x] refactored
  • [x] tested
• test_mnist_simple.py
  • [x] refactored
  • [ ] tested
• test_model_compilednetwork.py
  • [x] refactored
  • [ ] tested
• test_models.py
  • [x] refactored
  • [ ] tested
• test_network_custom_2d.py
  • [x] refactored
  • [x] tested
• test_network_custom_input_layers.py
  • [x] refactored
  • [x] tested
• test_network_custom_multiple_inputs.py
  • [x] refactored
  • [x] tested
• test_network_custom_multiple_outputs.py
  • [x] refactored
  • [x] tested
• test_network_sequential_1d.py
  • [x] refactored
  • [x] tested
• test_network_sequential_2d.py
  • [x] refactored
  • [x] tested
• test_network_sequential_3d.py
  • [x] refactored
  • [x] tested
• test_network_sequential_rnn.py
  • [x] refactored
  • [x] tested
• test_optimizer_amsgrad.py
  • [x] refactored
  • [ ] tested
• test_pydocstyle.py
  • [ ] refactored
  • [ ] tested
• test_reuse_mlp.py
  • [x] refactored
  • [ ] tested
• test_tf_layers.py
  • [x] refactored
  • [ ] tested
• test_timeout.py
  • [x] refactored
  • [x] tested
• test_utils_predict.py
  • [x] refactored
  • [ ] tested
• test_yapf_format.py
  • [x] refactored
  • [x] tested

Work to be done

Layers

• contrib
  • ROIPoolingLayer:
  • [x] refactored
  • [ ] tested
• convolution/
  • AtrousConv1dLayer:
  • [x] refactored => AtrousConv1dLayer and AtrousConv2dLayer are removed, use Conv1d/2d with dilation_rate instead.
  • [x] tested
• spatial_transformer.py
  • SpatialTransformer2dAffineLayer: see test_layers_spatial_transformer.py
  • [x] refactored
  • [x] tested
• stack.py
  • StackLayer:
  • [x] refactored
  • [ ] tested
  • UnStackLayer: => Need to be checked! Not working
  • [ ] refactored
  • [ ] tested
• time_distribution.py
  • TimeDistributedLayer:
  • [x] refactored
  • [ ] tested

TensorLayer Hub

• tl.models => became tl.hub
  • [ ] VGG16
  • [ ] VGG19
  • [ ] MobileNet
  • [ ] SqueezeNet

Examples

• basic_tutorials
  • [x] refactored
  • [ ] tested
• data_process
  • [x] refactored
  • [x] tested
• database
  • [x] refactored
  • [ ] tested
• deprecated_tutorials
  • [ ] refactored
  • [ ] tested
• distributed_training
  • [x] refactored
  • [ ] tested
• keras_tfslim
  • [x] refactored
  • [ ] tested
• pretrained_cnn
  • [x] refactored
  • [ ] tested
• quantized_net
  • [x] refactored
  • [ ] tested
• reinforcement_learning
  • [x] refactored
  • [ ] tested
• text_classification
  • [x] refactored
  • [ ] tested
• text_generation
  • [x] refactored
  • [ ] tested
• text_ptb
  • [x] refactored
  • [ ] tested
• text_word_embedding
  • [x] refactored
  • [ ] tested

More TODO

• [ ] add scope_name into the layer name

• [ ] don't check duplicated layer name in core layer

• Test code:

plh = tf.placeholder(tf.float16, (100, 32))
net = tl.layers.InputLayer(name='in')(plh)
with tf.variable_scope('test'):
    net = tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="dense")(net)
with tf.variable_scope('test', reuse=True):
    net = tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="dense")(net)
print(net['test/dense'])
print(net['test/dense_2'])
assert len(net.all_weights) == 2
print(net.all_params)  # give a warning, but still works

• How to do with manual-compile?:

with tf.variable_scope('test'):
    model.add(tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="seq_layer_9"))
with tf.variable_scope('test', reuse=True):
    model.add(tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="seq_layer_9"))
plh = tf.placeholder(tf.float16, (100, 32))
net = model.build(plc)
print(net['test/seq_layer_9'])

• Rename
  • [ ] assign_params --> assign_weights
  • [ ] print_params --> print_weights
  • [ ] print_layers --> print_outputs
  • [ ] all_layers --> all_outputs
  • [ ] all_params --> all_weights
  • [ ] local_weights
  • [ ] count_pararms --> count_all_weights
  • [ ] count_local_weights
  • [ ] restore_params --> restore_weights in tl.models/hub
  • [ ] get_layers_with_name --> get_outputs_with_name ?

Give exception and hint when users use the old name.

• Documentation

  • [ ] docs in python code
  • [ ] docs in RST

• Simplify layer name

  • [x] test folder
  • [x] layer folder
  • [x] example folder
  • for example,DenseLayer --> Dense then add code:

    
    def DenseLayer(*args, **kwargs):
    raise Exception("DenseLayer(net, 100) --> Dense(100)(net)")

def BatchNormLayer(*args, **kwargs): raise Exception("BatchNormLayer(net, is_train=True, name='bn') --> BatchNorm(name='bn')(net, is_train=True)")

[zsdonghao]

https://github.com/tensorlayer/tensorlayer/issues/900