Hassle free own data input to Transformer

[abhishek-niranjan]

System information

• What is the top-level directory of the model you are using: /home/models/official/transformer
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Redhat 7.4
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.12
• Bazel version (if compiling from source): NA
• CUDA/cuDNN version: cuda-9.0
• GPU model and memory: Tesla P40 and 24 GB
• Exact command to reproduce: NA

Describe the problem

Is there a hassle free way to input your own data to transformer model, say train.src and train.tgt without taking the code-bits from data_download.py and modifying to make it work?

[hichiaty]

If you use the tensor2tensor version, you can create a very simple script to register your "problem" in almost no time as described in the docs. https://github.com/tensorflow/tensor2tensor/blob/master/docs/new_problem.md

[abhishek-niranjan]

@hichiaty , actually I want to modify the architecture a little and prefer to work on this version as I feel it's fairly easy to make changes here. Besides, I believe that this implementation should also have an easier way if you want to train the model on your own data.

[abhishek-niranjan]

@hichiaty , actually I want to modify the architecture a little and prefer to work on this version as I feel it's fairly easy to make changes here. Besides, I believe that this implementation should also have an easier way if you want to train the model on your own data.

[abhishek-niranjan]

UPDATE: I've fixed a script from the pieces of data_download.py to be able to run the model on our own data. Closing the issue.