Submission shows "Takendown" status

[jcblaisecruz02]

Hi!

We're submitting for the FLORES Small Track 2 and after configuring the handler to work for our model, both the local and integrated tests passed. Model upload finished without any problems as well, but when the model finished processing on dynabench, it immediately gave us a "takendown" status.

Here's some details on our submission:

• Submission name: wmt21-srph
• Model number: 236
• System used for submission: AWS p2.xlarge
• I also checked out the other issue that had a "takendown" tag (#93) and added in our other files in model_files in the setup_config.json file as suggested.

Any ideas why the model failed to evaluate? Thanks in advance!

[gwenzek]

Hi, I'm seeing the following stack trace in the logs:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/dist-packages/ts/service.py", line 100, in predict
     ret = self._entry_point(input_batch, self.context)
   File "/home/model-server/tmp/models/7b8eea2d40534878b6e9830342ab8fb0/handler.py", line 145, in handle
     input_data = _service.preprocess(data)
   File "/home/model-server/tmp/models/7b8eea2d40534878b6e9830342ab8fb0/handler.py", line 88, in preprocess
     input_data = self.sp.encode('[{}] [{}] '.format(ISO2M100[example['sourceLanguage']], ISO2M100[example['targetLanguage']]) + example['sourceText'], out_type=str)
 TypeError: bytearray indices must be integers or slices, not str

Since it's a type error, I think this should be visible by running the tests at the bottom of the reference handler: https://github.com/gwenzek/flores/blob/master/dynalab/handler.py

I think you should start from the above handler and try to put your model specific code inside the Handler class.

[jcblaisecruz02]

Ah, thanks for the catch. This handler looks very different from the template in the main Dynalab repo. I'll follow this one instead. Thanks for the quick response!