ValueError: Invalid reduction dimension 2 for input with 2 dimensions for translation model training after updating Ludwig

[farazk86]

Describe the bug Hi, I previously used Ludwig when it was using the Tensorflow1.x backend. And I created a machine translation project using it. But now after updating ludwig to the latest version, I can no longer run the same project.

I am basically following the configuration mentioned in the example for translation: https://ludwig-ai.github.io/ludwig-docs/examples/#machine-translation

below is my model_definition.yaml

training:
    epochs: 500
    early_stop: 50
    batch_size: 128
#    dropout_rate: 0.3

input_features:
    -
        name: column1
        type: text
        level: word
        encoder: parallel_cnn
        representation: sparse      
        reduce_output: null
        preprocessing:
            word_tokenizer: space

output_features:
    -
        name: column2
        type: text
        level: word
        decoder: generator
        cell_type: lstm
        attention: bahdanau
        loss:
            type: sampled_softmax_cross_entropy

my terminal command:

ludwig train --experiment_name translate-1 --dataset training_file.csv --config_file model_definition.yaml --output_directory results

This is the error that is produced:

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,256], [] and with computed input tensors: input[1] = <2>.

Here are a few lines from the training_file.csv

column1,column2
k k klk k hjkj hg k kg h k jlk k kj kg hk k k k k k k k klk kjh jkj hg ghk kj kh khgh hg,N S SHL S LHHL LL H SL H H LHL S SL HL HH S S S S S S S SHL SLL HHL LL SHH SL HL HLLH SL
hk lk kh klk l lmlk lml mn m klm mn m ml lj kl klk kjhj jh h h h klm l lm l l l l lk lmkl lkk hjh h h klk kj k klm mlkj kl ml lk lk m lk jkjh jh k k hkh hg hk lm kj gh hg hjk jh,NH HL SL HHL H SHLL HHL HH L LHH SH L SL SL HH LHL SLLH SL S S S HHH L SH L S S S SL HHLH SLS LHL S S HHL SL H SHH SLLL HH HL SL HL H LL LHLL HL H S LHL SL HH HH LL LH SL HHH LL
kj klkjkjh ghg hj j j jh jk hj ghjh hg g fg g g g hjhg hjh gf gh hkjklkjh hjhg hg,NL HHLLHLL LHL HH S S SL HH LH LHHL SL S LH S S S HHLL HHL LL HH SHLHHLLL SHLL HL
g j k l k h k k g g g k k kj g hk k kj h kj h g g,N H H H L L H S L S S H S SL L HH S SL L HL L L S
hkj k k k k kkk kh kl kmlkjk kj h hl l l lk kmlm jlkk j hl l l lk k k kmlm k k k kmlk k k k k kml k kkk hjkjhj jh jkl ljl lmlkj kjhg hg kk hk h kkkh jkl lmlk lkj,NHL H S S S SSS SL HH LHLLLH SL L SH S S SL SHLH LHLS L LH S S SL S S SHLH L S S SHLL S S S S SHL L SSS LHHLLH SL HHH SLH SHLLL HLLL HL HS LH L HSSL HHH SHLL HLL
e e e e de dcded edb cb dc de dc bcdc cb c ac c c bcdcb d dededc bc dcbc ba,N S S S LH LLHHL HLL HL HL HH LL LHHL SL H LH S S LHHLL H SHLHLL LH HLLH LL
g j k l l l k h l k j g h g f gh h k k jk h h g g,N H H H S S L L H L L L H L L HH S H S LH L S L S
f fedf d dfe fg g g ggf ed df ef d dhj h hgf g f efg fe de dd c ed d cd d df ghgfg e,N SLLH L SHL HH S S SSL LL SH LH L SHH L SLL H L LHH LL LH LS L HL S LH S SH HHLLH L
d fgh g g g g gh g g g ge g fgh fe dfc de fefed dc f ghg g gh g g gh g ghkjh jkjh g ghjhgh hg h gf g h kl kjkl lk h gf hkj klk kjh jh hg hg gh g g g fgh fe dfc de fefed g ghgf gh g ghkjh jkjh ghjhgh hg,N HHH L S S S SH L S S SL H LHH LL LHL HH HLHLL SL H HHL S SH L S SH L SHHLL HHLL L SHHLLH SL H LL H H HH LLHH SL L LL HHL HHL SLL HL SL HL SH L S S LHH LL LHL HH HLHLL H SHLL HH L SHHLL HHLL LHHLLH SL

Environment (please complete the following information):

• OS: Windows 10
  • Python version 3.6.8
  • Ludwig version 0.33

[jimthompson5802]

@farazk86 thank you for submitting the issue. With the information you provided, I was able to recreate the issue on my end.

[farazk86]

Hi,

Thanks for verifying the bug. Is there any update on this? Is the problem related to Tensorflow 2? Would going to an earlier version of ludwig fix this?

[jimthompson5802]

@farazk86 re: the sampled_softmax issue. We are still looking at it. In the meantime, you should be able to use the regular softmax cross entropy loss function.

[farazk86]

@farazk86 re: the sampled_softmax issue. We are still looking at it. In the meantime, you should be able to use the regular softmax cross entropy loss function.

I am still getting the same error with softmax_xross_entropy :(

below is my model file:

training:
    epochs: 800
    early_stop: 60
    batch_size: 128
#    dropout_rate: 0.3

input_features:
    -
        name: nhl
        type: text
        level: word
        encoder: t5
        reduce_output: null
        trainable: true

output_features:
    -
        name: volpiano
        type: text
        level: word
        decoder: generator
        attention: bahdanau
        loss:
            type: softmax_cross_entropy

error:

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,512], [] and with computed input tensors: input[1] = <2>.

Please advise. :(

[jenishah]

Hi @jimthompson5802 , I admire this library and I would like to contribute.

I think https://github.com/ludwig-ai/ludwig/blob/9f03752c093949b6216f4a004549e0f0865fa442/ludwig/features/base_feature.py#L474

is causing the error. I tried by commenting out while keeping the loss type softmax_cross_entropy and it worked. I am not sure if that could have undesirable effects.

If you think I can work on this issue, please let me know.

[jimthompson5802]

@jenishah thank you for the positive feedback on Ludwig.

Re: the issue you are reporting. Please provide a the config.yaml and a small dataset, if possible, to see if I can reproduce the error.

[farazk86]

Hi @jimthompson5802 , I admire this library and I would like to contribute.

I think

https://github.com/ludwig-ai/ludwig/blob/9f03752c093949b6216f4a004549e0f0865fa442/ludwig/features/base_feature.py#L474

is causing the error. I tried by commenting out while keeping the loss type softmax_cross_entropy and it worked. I am not sure if that could have undesirable effects.

If you think I can work on this issue, please let me know.

Thanks for identifying the cause of this error :)

Did you find any effect of commenting this line out on your final result?

[jimthompson5802]

@farazk86 @jenishah I forgot to mention a PR was recently merged that fixed the sampled_softmax issue. This fix will be in the next release of Ludwig. In the meantime, if you want to see if the fix addresses this issue, just uninstall Ludwig and reinstall from Ludwig's main branch using this command:

pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this helped or not.

[farazk86]

@farazk86 @jenishah I forgot to mention a PR was recently merged that fixed the sampled_softmax issue. This fix will be in the next release of Ludwig. In the meantime, if you want to see if the fix addresses this issue, just uninstall Ludwig and reinstall from Ludwig's main branch using this command:

pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this helped or not.

Hi Jim,

Thanks for the reply. I was excited that the problem was resolved and uninstalled previous ludwig and installed from the above location as mentioned, but unfortunately I am getting the same error as before for the same dataset in my first post :(

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,512], [] and with computed input tensors: input[1] = <2>.

I tried with sampled_softmax_cross_entropy and softmax_cross_entropy.

by installing from git link my ludwig version is : 0.4.dev0

[jimthompson5802]

@farazk86 thank you for the feedback. Looks like there may be another subtle issue still left over from the latest PR. Let me take a look at it.

[jenishah]

@farazk86 @jenishah I forgot to mention a PR was recently merged that fixed the sampled_softmax issue. This fix will be in the next release of Ludwig. In the meantime, if you want to see if the fix addresses this issue, just uninstall Ludwig and reinstall from Ludwig's main branch using this command:

pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this helped or not.

Hi Jim,

Thanks for the reply. I was excited that the problem was resolved and uninstalled previous ludwig and installed from the above location as mentioned, but unfortunately I am getting the same error as before for the same dataset in my first post :(

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,512], [] and with computed input tensors: input[1] = <2>.

I tried with sampled_softmax_cross_entropy and softmax_cross_entropy.

by installing from git link my ludwig version is : 0.4.dev0

I was able to run the code with that.. But I did not get chance to look deep enough and verify if it is safe to add flag to skip that part.

[farazk86]

@farazk86 @jenishah I forgot to mention a PR was recently merged that fixed the sampled_softmax issue. This fix will be in the next release of Ludwig. In the meantime, if you want to see if the fix addresses this issue, just uninstall Ludwig and reinstall from Ludwig's main branch using this command:

pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this helped or not.

Hi Jim, Thanks for the reply. I was excited that the problem was resolved and uninstalled previous ludwig and installed from the above location as mentioned, but unfortunately I am getting the same error as before for the same dataset in my first post :(

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,512], [] and with computed input tensors: input[1] = <2>.

I tried with sampled_softmax_cross_entropy and softmax_cross_entropy. by installing from git link my ludwig version is : 0.4.dev0

I was able to run the code with that.. But I did not get chance to look deep enough and verify if it is safe to add flag to skip that part.

Hi @jenishah did you manage to run the code with the line of code you refered to earlier commented out or did your installation of github run on my data without any modification to the code?

Thanks

[jenishah]

@farazk86 @jenishah I forgot to mention a PR was recently merged that fixed the sampled_softmax issue. This fix will be in the next release of Ludwig. In the meantime, if you want to see if the fix addresses this issue, just uninstall Ludwig and reinstall from Ludwig's main branch using this command:

pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this helped or not.

Hi Jim, Thanks for the reply. I was excited that the problem was resolved and uninstalled previous ludwig and installed from the above location as mentioned, but unfortunately I am getting the same error as before for the same dataset in my first post :(

 ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_featu
re/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [128,512], [] and with computed input tensors: input[1] = <2>.

I tried with sampled_softmax_cross_entropy and softmax_cross_entropy. by installing from git link my ludwig version is : 0.4.dev0

I was able to run the code with that.. But I did not get chance to look deep enough and verify if it is safe to add flag to skip that part.

Hi @jenishah did you manage to run the code with the line of code you refered to earlier commented out or did your installation of github run on my data without any modification to the code?

Thanks

I ran on your data, and your yaml file, but I commented out the line that I mentioned. Also, it worked only with softmax_cross_entropy loss

[w4nderlust]

We are looking into this issue anyway, will update soon.

[Losbal]

We are looking into this issue anyway, will update soon.

Have there been any updates on the reduce input error? I am also experiencing the same issue and receiving the same error reported above:

ValueError: Invalid reduction dimension 2 for input with 2 dimensions. for '{{node ecd/text_output_feature/Max}} = Max[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false](ecd/text_output_feature/Abs_1, ecd/text_output_feature/Max/reduction_indices)' with input shapes: [64,512], [] and with computed input tensors: input[1] = <2>.

Thanks in advance for your help!

[w4nderlust]

@Losbal just to confirm, this still happens in the latest release v0.4.1? What if you tried using the current master (v0.5-dev0)? We have done a big rework on master so it's very likely that the issue was solved as a side effect of it. If not @justinxzhao could potentially help.

[Losbal]

@w4nderlust Yes, I was using v0.4.1 but I just installed v0.5 on a separate cluster to test and it seems to have resolved the issue. Thanks for your help! (N.B. I mentioned a different KeyError after switching to v0.5 in an earlier version of this reply but was able to resolve it, so please disregard it if you caught it pre-edit).

[justinxzhao]

Thanks for the update @Losbal. Glad you aren't running into any errors on 0.5. Closing, though feel free to reopen if you run into any issues.

[w4nderlust]

@farazk86 @Losbal sorry it took so long to solve the issue, glad we managed to do it though! :)