search

dsindex / etagger

reference tensorflow code for named entity tagging
107 stars 22 forks source link

Performance issue in input.py (by P3) #7

Open DLPerf opened 3 years ago

DLPerf commented 3 years ago

Hello! I've found a performance issue in input.py: dataset.batch(batch_size)(line 143) should be called before dataset.map(parser,num_parallel_calls=tf.data.experimental.AUTOTUNE)(line 141), which could make your program more efficient.

Here is the tensorflow document to support it.

Besides, you need to check the function parser called in dataset.map(parser,num_parallel_calls=tf.data.experimental.AUTOTUNE) whether to be affected or not to make the changed code work properly. For example, if parser needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z) after fix.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

dsindex commented 3 years ago

good point you addressed. i’m looking for PR :) but i think the modification for ‘parser’ to accept data with batch dimension, a little bit mesh. so, i’m worried about reducing readability.