search

allanj / ner_incomplete_annotation

129 stars 33 forks source link

batch_size #5

Closed yangshao closed 4 years ago

yangshao commented 4 years ago

Hi, Does the dynet vesion support batch size > 1?

allanj commented 4 years ago

Sorry. The dynet version currently only support batch size of 1. We tested the batch size > 1 with auto-batching mechanism in Dynet. But it seems the speed is not even accelerating with the existence of CRF layer.

yangshao commented 4 years ago

@allanj thanks for the reply, for the new pytorch version, i notice the batch size is supported, if i want to use a larger batch size like 128, which learning rate do i need to use as i see the loss is sum of batch instead of mean of batch, and why the default epoch changes to 100 instead of 30 in the dynet version?

allanj commented 4 years ago

Sorry for the late reply.

  1. So if you are using a larger batch size, you may still use 0.01 learning rate first and see you need to decay them. You might need some hyperparameter searching.
  2. Because the dynet version is with batch size 1, 30 epochs already include many updates. While with batch size of 10 in the Pytorch version, we can afford to run more epochs and each epoch has fewer updates.