Weird eigen error with concatenate_cols

[pmichel31415]

So this is a hard one to reproduce because it only happens to me in a big computation graph in a attentional seq2seq model but when running this code:

encoded_states = self.es.transduce(wembs)     # <-- transduce with vanilla_lstm
H_fwd = dy.concatenate_cols(encoded_states) # <-- breaks here

I get this eigen error with immediate_computation (without it just crashes with a python segfault)

Eigen::TensorEvaluator<const Eigen::TensorSlicingOp<StartIndices, Sizes, XprType>, Device>::TensorEvaluator(const XprType&, const Device&) [with StartIndices = const Eigen::DSizes<long int, 5>; Sizes = const Eigen::DSizes<long int, 5>; ArgType = Eigen::TensorMap<Eigen::Tensor<float, 5> >; Device = Eigen::DefaultDevice; Eigen::TensorEvaluator<const Eigen::TensorSlicingOp<StartIndices, Sizes, XprType>, Device>::XprType = Eigen::TensorSlicingOp<const Eigen::DSizes<long int, 5>, const Eigen::DSizes<long int, 5>, Eigen::TensorMap<Eigen::Tensor<float, 5> > >]: Assertion `m_impl.dimensions()[i] >= op.sizes()[i] + op.startIndices()[i]' failed.

I'm guessing this happens at this line. I've confirmed my inputs have matching dimensions (512 of batch size 32).

This happens on CPU/GPU/with or without autobatching.

This don't think this happened with the 2.0 release but I could be wrong, I'll update the issue once I test it on 2.0

If anyone knows more about what could cause this issue it would be very helpful!

[neubig]

Can you run dy.print_text_graphviz() for the graph that is causing the error, and also pinpoint the node that causes it to crash?

[pmichel31415]

The node is Concatenate.

I also confirmed that the bug didn't happen on the 2.0 release.

I'm not sure that print_graphviz would help because the program crashes upon calling concatenate even without autobatching (so maybe this is not a problem with the forward pass actually but with the graph itself or the python bindings... ?).

Anyway here is the result (called after the call to dy.concatenate_cols with the 2.0 version (so no bug)):

digraph G {
  rankdir=LR;
  nodesep=.05;
  N0 [label="v0 = parameters({2048,512}) @ 0xa6010c0"];
  N1 [label="v1 = parameters({2048,512}) @ 0xa5fd740"];
  N2 [label="v2 = parameters({2048}) @ 0xa5ff1a0"];
  N3 [label="v3 = random_bernoulli({512}, 0.6)"];
  N4 [label="v4 = random_bernoulli({512}, 1)"];
  N5 [label="v5 = random_bernoulli({512X32}, 0.6)"];
  N6 [label="v6 = random_bernoulli({512X32}, 1)"];
  N7 [label="v7 = parameters({2048,512}) @ 0xa5ff670"];
  N8 [label="v8 = parameters({2048,512}) @ 0xa5ff7b0"];
  N9 [label="v9 = parameters({2048}) @ 0xa5ff940"];
  N10 [label="v10 = random_bernoulli({512}, 0.6)"];
  N11 [label="v11 = random_bernoulli({512}, 1)"];
  N12 [label="v12 = random_bernoulli({512X32}, 0.6)"];
  N13 [label="v13 = random_bernoulli({512X32}, 1)"];
  N14 [label="v14 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N15 [label="v15 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N16 [label="v16 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N17 [label="v17 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N18 [label="v18 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N19 [label="v19 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N20 [label="v20 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N21 [label="v21 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N22 [label="v22 = lookup_parameters(|x|=39742 --> {512X32}) @ 0xa5ff290"];
  N23 [label="v23 = v14 \cdot v5"];
  N14 -> N23;
  N5 -> N23;
  N24 [label="v24 = v2 + v0 * v23"];
  N2 -> N24;
  N0 -> N24;
  N23 -> N24;
  N25 [label="v25 = slice(v24,0:512, dim=0)"];
  N24 -> N25;
  N26 [label="v26 = slice(v24,512:1024, dim=0)"];
  N24 -> N26;
  N27 [label="v27 = slice(v24,1024:1536, dim=0)"];
  N24 -> N27;
  N28 [label="v28 = slice(v24,1536:2048, dim=0)"];
  N24 -> N28;
  N29 [label="v29 = \sigma(v25)"];
  N25 -> N29;
  N30 [label="v30 = 1 + v26"];
  N26 -> N30;
  N31 [label="v31 = \sigma(v30)"];
  N30 -> N31;
  N32 [label="v32 = \sigma(v27)"];
  N27 -> N32;
  N33 [label="v33 = tanh(v28)"];
  N28 -> N33;
  N34 [label="v34 = v29 \cdot v33"];
  N29 -> N34;
  N33 -> N34;
  N35 [label="v35 = tanh(v34)"];
  N34 -> N35;
  N36 [label="v36 = v32 \cdot v35"];
  N32 -> N36;
  N35 -> N36;
  N37 [label="v37 = v15 \cdot v5"];
  N15 -> N37;
  N5 -> N37;
  N38 [label="v38 = v2 + v0 * v37 + v1 * v36"];
  N2 -> N38;
  N0 -> N38;
  N37 -> N38;
  N1 -> N38;
  N36 -> N38;
  N39 [label="v39 = slice(v38,0:512, dim=0)"];
  N38 -> N39;
  N40 [label="v40 = slice(v38,512:1024, dim=0)"];
  N38 -> N40;
  N41 [label="v41 = slice(v38,1024:1536, dim=0)"];
  N38 -> N41;
  N42 [label="v42 = slice(v38,1536:2048, dim=0)"];
  N38 -> N42;
  N43 [label="v43 = \sigma(v39)"];
  N39 -> N43;
  N44 [label="v44 = 1 + v40"];
  N40 -> N44;
  N45 [label="v45 = \sigma(v44)"];
  N44 -> N45;
  N46 [label="v46 = \sigma(v41)"];
  N41 -> N46;
  N47 [label="v47 = tanh(v42)"];
  N42 -> N47;
  N48 [label="v48 = v43 \cdot v47"];
  N43 -> N48;
  N47 -> N48;
  N49 [label="v49 = v45 \cdot v34"];
  N45 -> N49;
  N34 -> N49;
  N50 [label="v50 = v49 + v48"];
  N49 -> N50;
  N48 -> N50;
  N51 [label="v51 = tanh(v50)"];
  N50 -> N51;
  N52 [label="v52 = v46 \cdot v51"];
  N46 -> N52;
  N51 -> N52;
  N53 [label="v53 = v16 \cdot v5"];
  N16 -> N53;
  N5 -> N53;
  N54 [label="v54 = v2 + v0 * v53 + v1 * v52"];
  N2 -> N54;
  N0 -> N54;
  N53 -> N54;
  N1 -> N54;
  N52 -> N54;
  N55 [label="v55 = slice(v54,0:512, dim=0)"];
  N54 -> N55;
  N56 [label="v56 = slice(v54,512:1024, dim=0)"];
  N54 -> N56;
  N57 [label="v57 = slice(v54,1024:1536, dim=0)"];
  N54 -> N57;
  N58 [label="v58 = slice(v54,1536:2048, dim=0)"];
  N54 -> N58;
  N59 [label="v59 = \sigma(v55)"];
  N55 -> N59;
  N60 [label="v60 = 1 + v56"];
  N56 -> N60;
  N61 [label="v61 = \sigma(v60)"];
  N60 -> N61;
  N62 [label="v62 = \sigma(v57)"];
  N57 -> N62;
  N63 [label="v63 = tanh(v58)"];
  N58 -> N63;
  N64 [label="v64 = v59 \cdot v63"];
  N59 -> N64;
  N63 -> N64;
  N65 [label="v65 = v61 \cdot v50"];
  N61 -> N65;
  N50 -> N65;
  N66 [label="v66 = v65 + v64"];
  N65 -> N66;
  N64 -> N66;
  N67 [label="v67 = tanh(v66)"];
  N66 -> N67;
  N68 [label="v68 = v62 \cdot v67"];
  N62 -> N68;
  N67 -> N68;
  N69 [label="v69 = v17 \cdot v5"];
  N17 -> N69;
  N5 -> N69;
  N70 [label="v70 = v2 + v0 * v69 + v1 * v68"];
  N2 -> N70;
  N0 -> N70;
  N69 -> N70;
  N1 -> N70;
  N68 -> N70;
  N71 [label="v71 = slice(v70,0:512, dim=0)"];
  N70 -> N71;
  N72 [label="v72 = slice(v70,512:1024, dim=0)"];
  N70 -> N72;
  N73 [label="v73 = slice(v70,1024:1536, dim=0)"];
  N70 -> N73;
  N74 [label="v74 = slice(v70,1536:2048, dim=0)"];
  N70 -> N74;
  N75 [label="v75 = \sigma(v71)"];
  N71 -> N75;
  N76 [label="v76 = 1 + v72"];
  N72 -> N76;
  N77 [label="v77 = \sigma(v76)"];
  N76 -> N77;
  N78 [label="v78 = \sigma(v73)"];
  N73 -> N78;
  N79 [label="v79 = tanh(v74)"];
  N74 -> N79;
  N80 [label="v80 = v75 \cdot v79"];
  N75 -> N80;
  N79 -> N80;
  N81 [label="v81 = v77 \cdot v66"];
  N77 -> N81;
  N66 -> N81;
  N82 [label="v82 = v81 + v80"];
  N81 -> N82;
  N80 -> N82;
  N83 [label="v83 = tanh(v82)"];
  N82 -> N83;
  N84 [label="v84 = v78 \cdot v83"];
  N78 -> N84;
  N83 -> N84;
  N85 [label="v85 = v18 \cdot v5"];
  N18 -> N85;
  N5 -> N85;
  N86 [label="v86 = v2 + v0 * v85 + v1 * v84"];
  N2 -> N86;
  N0 -> N86;
  N85 -> N86;
  N1 -> N86;
  N84 -> N86;
  N87 [label="v87 = slice(v86,0:512, dim=0)"];
  N86 -> N87;
  N88 [label="v88 = slice(v86,512:1024, dim=0)"];
  N86 -> N88;
  N89 [label="v89 = slice(v86,1024:1536, dim=0)"];
  N86 -> N89;
  N90 [label="v90 = slice(v86,1536:2048, dim=0)"];
  N86 -> N90;
  N91 [label="v91 = \sigma(v87)"];
  N87 -> N91;
  N92 [label="v92 = 1 + v88"];
  N88 -> N92;
  N93 [label="v93 = \sigma(v92)"];
  N92 -> N93;
  N94 [label="v94 = \sigma(v89)"];
  N89 -> N94;
  N95 [label="v95 = tanh(v90)"];
  N90 -> N95;
  N96 [label="v96 = v91 \cdot v95"];
  N91 -> N96;
  N95 -> N96;
  N97 [label="v97 = v93 \cdot v82"];
  N93 -> N97;
  N82 -> N97;
  N98 [label="v98 = v97 + v96"];
  N97 -> N98;
  N96 -> N98;
  N99 [label="v99 = tanh(v98)"];
  N98 -> N99;
  N100 [label="v100 = v94 \cdot v99"];
  N94 -> N100;
  N99 -> N100;
  N101 [label="v101 = v19 \cdot v5"];
  N19 -> N101;
  N5 -> N101;
  N102 [label="v102 = v2 + v0 * v101 + v1 * v100"];
  N2 -> N102;
  N0 -> N102;
  N101 -> N102;
  N1 -> N102;
  N100 -> N102;
  N103 [label="v103 = slice(v102,0:512, dim=0)"];
  N102 -> N103;
  N104 [label="v104 = slice(v102,512:1024, dim=0)"];
  N102 -> N104;
  N105 [label="v105 = slice(v102,1024:1536, dim=0)"];
  N102 -> N105;
  N106 [label="v106 = slice(v102,1536:2048, dim=0)"];
  N102 -> N106;
  N107 [label="v107 = \sigma(v103)"];
  N103 -> N107;
  N108 [label="v108 = 1 + v104"];
  N104 -> N108;
  N109 [label="v109 = \sigma(v108)"];
  N108 -> N109;
  N110 [label="v110 = \sigma(v105)"];
  N105 -> N110;
  N111 [label="v111 = tanh(v106)"];
  N106 -> N111;
  N112 [label="v112 = v107 \cdot v111"];
  N107 -> N112;
  N111 -> N112;
  N113 [label="v113 = v109 \cdot v98"];
  N109 -> N113;
  N98 -> N113;
  N114 [label="v114 = v113 + v112"];
  N113 -> N114;
  N112 -> N114;
  N115 [label="v115 = tanh(v114)"];
  N114 -> N115;
  N116 [label="v116 = v110 \cdot v115"];
  N110 -> N116;
  N115 -> N116;
  N117 [label="v117 = v20 \cdot v5"];
  N20 -> N117;
  N5 -> N117;
  N118 [label="v118 = v2 + v0 * v117 + v1 * v116"];
  N2 -> N118;
  N0 -> N118;
  N117 -> N118;
  N1 -> N118;
  N116 -> N118;
  N119 [label="v119 = slice(v118,0:512, dim=0)"];
  N118 -> N119;
  N120 [label="v120 = slice(v118,512:1024, dim=0)"];
  N118 -> N120;
  N121 [label="v121 = slice(v118,1024:1536, dim=0)"];
  N118 -> N121;
  N122 [label="v122 = slice(v118,1536:2048, dim=0)"];
  N118 -> N122;
  N123 [label="v123 = \sigma(v119)"];
  N119 -> N123;
  N124 [label="v124 = 1 + v120"];
  N120 -> N124;
  N125 [label="v125 = \sigma(v124)"];
  N124 -> N125;
  N126 [label="v126 = \sigma(v121)"];
  N121 -> N126;
  N127 [label="v127 = tanh(v122)"];
  N122 -> N127;
  N128 [label="v128 = v123 \cdot v127"];
  N123 -> N128;
  N127 -> N128;
  N129 [label="v129 = v125 \cdot v114"];
  N125 -> N129;
  N114 -> N129;
  N130 [label="v130 = v129 + v128"];
  N129 -> N130;
  N128 -> N130;
  N131 [label="v131 = tanh(v130)"];
  N130 -> N131;
  N132 [label="v132 = v126 \cdot v131"];
  N126 -> N132;
  N131 -> N132;
  N133 [label="v133 = v21 \cdot v5"];
  N21 -> N133;
  N5 -> N133;
  N134 [label="v134 = v2 + v0 * v133 + v1 * v132"];
  N2 -> N134;
  N0 -> N134;
  N133 -> N134;
  N1 -> N134;
  N132 -> N134;
  N135 [label="v135 = slice(v134,0:512, dim=0)"];
  N134 -> N135;
  N136 [label="v136 = slice(v134,512:1024, dim=0)"];
  N134 -> N136;
  N137 [label="v137 = slice(v134,1024:1536, dim=0)"];
  N134 -> N137;
  N138 [label="v138 = slice(v134,1536:2048, dim=0)"];
  N134 -> N138;
  N139 [label="v139 = \sigma(v135)"];
  N135 -> N139;
  N140 [label="v140 = 1 + v136"];
  N136 -> N140;
  N141 [label="v141 = \sigma(v140)"];
  N140 -> N141;
  N142 [label="v142 = \sigma(v137)"];
  N137 -> N142;
  N143 [label="v143 = tanh(v138)"];
  N138 -> N143;
  N144 [label="v144 = v139 \cdot v143"];
  N139 -> N144;
  N143 -> N144;
  N145 [label="v145 = v141 \cdot v130"];
  N141 -> N145;
  N130 -> N145;
  N146 [label="v146 = v145 + v144"];
  N145 -> N146;
  N144 -> N146;
  N147 [label="v147 = tanh(v146)"];
  N146 -> N147;
  N148 [label="v148 = v142 \cdot v147"];
  N142 -> N148;
  N147 -> N148;
  N149 [label="v149 = v22 \cdot v5"];
  N22 -> N149;
  N5 -> N149;
  N150 [label="v150 = v2 + v0 * v149 + v1 * v148"];
  N2 -> N150;
  N0 -> N150;
  N149 -> N150;
  N1 -> N150;
  N148 -> N150;
  N151 [label="v151 = slice(v150,0:512, dim=0)"];
  N150 -> N151;
  N152 [label="v152 = slice(v150,512:1024, dim=0)"];
  N150 -> N152;
  N153 [label="v153 = slice(v150,1024:1536, dim=0)"];
  N150 -> N153;
  N154 [label="v154 = slice(v150,1536:2048, dim=0)"];
  N150 -> N154;
  N155 [label="v155 = \sigma(v151)"];
  N151 -> N155;
  N156 [label="v156 = 1 + v152"];
  N152 -> N156;
  N157 [label="v157 = \sigma(v156)"];
  N156 -> N157;
  N158 [label="v158 = \sigma(v153)"];
  N153 -> N158;
  N159 [label="v159 = tanh(v154)"];
  N154 -> N159;
  N160 [label="v160 = v155 \cdot v159"];
  N155 -> N160;
  N159 -> N160;
  N161 [label="v161 = v157 \cdot v146"];
  N157 -> N161;
  N146 -> N161;
  N162 [label="v162 = v161 + v160"];
  N161 -> N162;
  N160 -> N162;
  N163 [label="v163 = tanh(v162)"];
  N162 -> N163;
  N164 [label="v164 = v158 \cdot v163"];
  N158 -> N164;
  N163 -> N164;
  N165 [label="v165 = v22 \cdot v12"];
  N22 -> N165;
  N12 -> N165;
  N166 [label="v166 = v9 + v7 * v165"];
  N9 -> N166;
  N7 -> N166;
  N165 -> N166;
  N167 [label="v167 = slice(v166,0:512, dim=0)"];
  N166 -> N167;
  N168 [label="v168 = slice(v166,512:1024, dim=0)"];
  N166 -> N168;
  N169 [label="v169 = slice(v166,1024:1536, dim=0)"];
  N166 -> N169;
  N170 [label="v170 = slice(v166,1536:2048, dim=0)"];
  N166 -> N170;
  N171 [label="v171 = \sigma(v167)"];
  N167 -> N171;
  N172 [label="v172 = 1 + v168"];
  N168 -> N172;
  N173 [label="v173 = \sigma(v172)"];
  N172 -> N173;
  N174 [label="v174 = \sigma(v169)"];
  N169 -> N174;
  N175 [label="v175 = tanh(v170)"];
  N170 -> N175;
  N176 [label="v176 = v171 \cdot v175"];
  N171 -> N176;
  N175 -> N176;
  N177 [label="v177 = tanh(v176)"];
  N176 -> N177;
  N178 [label="v178 = v174 \cdot v177"];
  N174 -> N178;
  N177 -> N178;
  N179 [label="v179 = v21 \cdot v12"];
  N21 -> N179;
  N12 -> N179;
  N180 [label="v180 = v9 + v7 * v179 + v8 * v178"];
  N9 -> N180;
  N7 -> N180;
  N179 -> N180;
  N8 -> N180;
  N178 -> N180;
  N181 [label="v181 = slice(v180,0:512, dim=0)"];
  N180 -> N181;
  N182 [label="v182 = slice(v180,512:1024, dim=0)"];
  N180 -> N182;
  N183 [label="v183 = slice(v180,1024:1536, dim=0)"];
  N180 -> N183;
  N184 [label="v184 = slice(v180,1536:2048, dim=0)"];
  N180 -> N184;
  N185 [label="v185 = \sigma(v181)"];
  N181 -> N185;
  N186 [label="v186 = 1 + v182"];
  N182 -> N186;
  N187 [label="v187 = \sigma(v186)"];
  N186 -> N187;
  N188 [label="v188 = \sigma(v183)"];
  N183 -> N188;
  N189 [label="v189 = tanh(v184)"];
  N184 -> N189;
  N190 [label="v190 = v185 \cdot v189"];
  N185 -> N190;
  N189 -> N190;
  N191 [label="v191 = v187 \cdot v176"];
  N187 -> N191;
  N176 -> N191;
  N192 [label="v192 = v191 + v190"];
  N191 -> N192;
  N190 -> N192;
  N193 [label="v193 = tanh(v192)"];
  N192 -> N193;
  N194 [label="v194 = v188 \cdot v193"];
  N188 -> N194;
  N193 -> N194;
  N195 [label="v195 = v20 \cdot v12"];
  N20 -> N195;
  N12 -> N195;
  N196 [label="v196 = v9 + v7 * v195 + v8 * v194"];
  N9 -> N196;
  N7 -> N196;
  N195 -> N196;
  N8 -> N196;
  N194 -> N196;
  N197 [label="v197 = slice(v196,0:512, dim=0)"];
  N196 -> N197;
  N198 [label="v198 = slice(v196,512:1024, dim=0)"];
  N196 -> N198;
  N199 [label="v199 = slice(v196,1024:1536, dim=0)"];
  N196 -> N199;
  N200 [label="v200 = slice(v196,1536:2048, dim=0)"];
  N196 -> N200;
  N201 [label="v201 = \sigma(v197)"];
  N197 -> N201;
  N202 [label="v202 = 1 + v198"];
  N198 -> N202;
  N203 [label="v203 = \sigma(v202)"];
  N202 -> N203;
  N204 [label="v204 = \sigma(v199)"];
  N199 -> N204;
  N205 [label="v205 = tanh(v200)"];
  N200 -> N205;
  N206 [label="v206 = v201 \cdot v205"];
  N201 -> N206;
  N205 -> N206;
  N207 [label="v207 = v203 \cdot v192"];
  N203 -> N207;
  N192 -> N207;
  N208 [label="v208 = v207 + v206"];
  N207 -> N208;
  N206 -> N208;
  N209 [label="v209 = tanh(v208)"];
  N208 -> N209;
  N210 [label="v210 = v204 \cdot v209"];
  N204 -> N210;
  N209 -> N210;
  N211 [label="v211 = v19 \cdot v12"];
  N19 -> N211;
  N12 -> N211;
  N212 [label="v212 = v9 + v7 * v211 + v8 * v210"];
  N9 -> N212;
  N7 -> N212;
  N211 -> N212;
  N8 -> N212;
  N210 -> N212;
  N213 [label="v213 = slice(v212,0:512, dim=0)"];
  N212 -> N213;
  N214 [label="v214 = slice(v212,512:1024, dim=0)"];
  N212 -> N214;
  N215 [label="v215 = slice(v212,1024:1536, dim=0)"];
  N212 -> N215;
  N216 [label="v216 = slice(v212,1536:2048, dim=0)"];
  N212 -> N216;
  N217 [label="v217 = \sigma(v213)"];
  N213 -> N217;
  N218 [label="v218 = 1 + v214"];
  N214 -> N218;
  N219 [label="v219 = \sigma(v218)"];
  N218 -> N219;
  N220 [label="v220 = \sigma(v215)"];
  N215 -> N220;
  N221 [label="v221 = tanh(v216)"];
  N216 -> N221;
  N222 [label="v222 = v217 \cdot v221"];
  N217 -> N222;
  N221 -> N222;
  N223 [label="v223 = v219 \cdot v208"];
  N219 -> N223;
  N208 -> N223;
  N224 [label="v224 = v223 + v222"];
  N223 -> N224;
  N222 -> N224;
  N225 [label="v225 = tanh(v224)"];
  N224 -> N225;
  N226 [label="v226 = v220 \cdot v225"];
  N220 -> N226;
  N225 -> N226;
  N227 [label="v227 = v18 \cdot v12"];
  N18 -> N227;
  N12 -> N227;
  N228 [label="v228 = v9 + v7 * v227 + v8 * v226"];
  N9 -> N228;
  N7 -> N228;
  N227 -> N228;
  N8 -> N228;
  N226 -> N228;
  N229 [label="v229 = slice(v228,0:512, dim=0)"];
  N228 -> N229;
  N230 [label="v230 = slice(v228,512:1024, dim=0)"];
  N228 -> N230;
  N231 [label="v231 = slice(v228,1024:1536, dim=0)"];
  N228 -> N231;
  N232 [label="v232 = slice(v228,1536:2048, dim=0)"];
  N228 -> N232;
  N233 [label="v233 = \sigma(v229)"];
  N229 -> N233;
  N234 [label="v234 = 1 + v230"];
  N230 -> N234;
  N235 [label="v235 = \sigma(v234)"];
  N234 -> N235;
  N236 [label="v236 = \sigma(v231)"];
  N231 -> N236;
  N237 [label="v237 = tanh(v232)"];
  N232 -> N237;
  N238 [label="v238 = v233 \cdot v237"];
  N233 -> N238;
  N237 -> N238;
  N239 [label="v239 = v235 \cdot v224"];
  N235 -> N239;
  N224 -> N239;
  N240 [label="v240 = v239 + v238"];
  N239 -> N240;
  N238 -> N240;
  N241 [label="v241 = tanh(v240)"];
  N240 -> N241;
  N242 [label="v242 = v236 \cdot v241"];
  N236 -> N242;
  N241 -> N242;
  N243 [label="v243 = v17 \cdot v12"];
  N17 -> N243;
  N12 -> N243;
  N244 [label="v244 = v9 + v7 * v243 + v8 * v242"];
  N9 -> N244;
  N7 -> N244;
  N243 -> N244;
  N8 -> N244;
  N242 -> N244;
  N245 [label="v245 = slice(v244,0:512, dim=0)"];
  N244 -> N245;
  N246 [label="v246 = slice(v244,512:1024, dim=0)"];
  N244 -> N246;
  N247 [label="v247 = slice(v244,1024:1536, dim=0)"];
  N244 -> N247;
  N248 [label="v248 = slice(v244,1536:2048, dim=0)"];
  N244 -> N248;
  N249 [label="v249 = \sigma(v245)"];
  N245 -> N249;
  N250 [label="v250 = 1 + v246"];
  N246 -> N250;
  N251 [label="v251 = \sigma(v250)"];
  N250 -> N251;
  N252 [label="v252 = \sigma(v247)"];
  N247 -> N252;
  N253 [label="v253 = tanh(v248)"];
  N248 -> N253;
  N254 [label="v254 = v249 \cdot v253"];
  N249 -> N254;
  N253 -> N254;
  N255 [label="v255 = v251 \cdot v240"];
  N251 -> N255;
  N240 -> N255;
  N256 [label="v256 = v255 + v254"];
  N255 -> N256;
  N254 -> N256;
  N257 [label="v257 = tanh(v256)"];
  N256 -> N257;
  N258 [label="v258 = v252 \cdot v257"];
  N252 -> N258;
  N257 -> N258;
  N259 [label="v259 = v16 \cdot v12"];
  N16 -> N259;
  N12 -> N259;
  N260 [label="v260 = v9 + v7 * v259 + v8 * v258"];
  N9 -> N260;
  N7 -> N260;
  N259 -> N260;
  N8 -> N260;
  N258 -> N260;
  N261 [label="v261 = slice(v260,0:512, dim=0)"];
  N260 -> N261;
  N262 [label="v262 = slice(v260,512:1024, dim=0)"];
  N260 -> N262;
  N263 [label="v263 = slice(v260,1024:1536, dim=0)"];
  N260 -> N263;
  N264 [label="v264 = slice(v260,1536:2048, dim=0)"];
  N260 -> N264;
  N265 [label="v265 = \sigma(v261)"];
  N261 -> N265;
  N266 [label="v266 = 1 + v262"];
  N262 -> N266;
  N267 [label="v267 = \sigma(v266)"];
  N266 -> N267;
  N268 [label="v268 = \sigma(v263)"];
  N263 -> N268;
  N269 [label="v269 = tanh(v264)"];
  N264 -> N269;
  N270 [label="v270 = v265 \cdot v269"];
  N265 -> N270;
  N269 -> N270;
  N271 [label="v271 = v267 \cdot v256"];
  N267 -> N271;
  N256 -> N271;
  N272 [label="v272 = v271 + v270"];
  N271 -> N272;
  N270 -> N272;
  N273 [label="v273 = tanh(v272)"];
  N272 -> N273;
  N274 [label="v274 = v268 \cdot v273"];
  N268 -> N274;
  N273 -> N274;
  N275 [label="v275 = v15 \cdot v12"];
  N15 -> N275;
  N12 -> N275;
  N276 [label="v276 = v9 + v7 * v275 + v8 * v274"];
  N9 -> N276;
  N7 -> N276;
  N275 -> N276;
  N8 -> N276;
  N274 -> N276;
  N277 [label="v277 = slice(v276,0:512, dim=0)"];
  N276 -> N277;
  N278 [label="v278 = slice(v276,512:1024, dim=0)"];
  N276 -> N278;
  N279 [label="v279 = slice(v276,1024:1536, dim=0)"];
  N276 -> N279;
  N280 [label="v280 = slice(v276,1536:2048, dim=0)"];
  N276 -> N280;
  N281 [label="v281 = \sigma(v277)"];
  N277 -> N281;
  N282 [label="v282 = 1 + v278"];
  N278 -> N282;
  N283 [label="v283 = \sigma(v282)"];
  N282 -> N283;
  N284 [label="v284 = \sigma(v279)"];
  N279 -> N284;
  N285 [label="v285 = tanh(v280)"];
  N280 -> N285;
  N286 [label="v286 = v281 \cdot v285"];
  N281 -> N286;
  N285 -> N286;
  N287 [label="v287 = v283 \cdot v272"];
  N283 -> N287;
  N272 -> N287;
  N288 [label="v288 = v287 + v286"];
  N287 -> N288;
  N286 -> N288;
  N289 [label="v289 = tanh(v288)"];
  N288 -> N289;
  N290 [label="v290 = v284 \cdot v289"];
  N284 -> N290;
  N289 -> N290;
  N291 [label="v291 = v14 \cdot v12"];
  N14 -> N291;
  N12 -> N291;
  N292 [label="v292 = v9 + v7 * v291 + v8 * v290"];
  N9 -> N292;
  N7 -> N292;
  N291 -> N292;
  N8 -> N292;
  N290 -> N292;
  N293 [label="v293 = slice(v292,0:512, dim=0)"];
  N292 -> N293;
  N294 [label="v294 = slice(v292,512:1024, dim=0)"];
  N292 -> N294;
  N295 [label="v295 = slice(v292,1024:1536, dim=0)"];
  N292 -> N295;
  N296 [label="v296 = slice(v292,1536:2048, dim=0)"];
  N292 -> N296;
  N297 [label="v297 = \sigma(v293)"];
  N293 -> N297;
  N298 [label="v298 = 1 + v294"];
  N294 -> N298;
  N299 [label="v299 = \sigma(v298)"];
  N298 -> N299;
  N300 [label="v300 = \sigma(v295)"];
  N295 -> N300;
  N301 [label="v301 = tanh(v296)"];
  N296 -> N301;
  N302 [label="v302 = v297 \cdot v301"];
  N297 -> N302;
  N301 -> N302;
  N303 [label="v303 = v299 \cdot v288"];
  N299 -> N303;
  N288 -> N303;
  N304 [label="v304 = v303 + v302"];
  N303 -> N304;
  N302 -> N304;
  N305 [label="v305 = tanh(v304)"];
  N304 -> N305;
  N306 [label="v306 = v300 \cdot v305"];
  N300 -> N306;
  N305 -> N306;
  N307 [label="v307 = || v164 ||^2"];
  N164 -> N307;
  N308 [label="v308 = moment_batches( expression=v307, order=1 )"];
  N307 -> N308;
  N309 [label="v309 = concat({v36,v52,v68,v84,v100,v116,v132,v148,v164}, 1)"];
  N36 -> N309;
  N52 -> N309;
  N68 -> N309;
  N84 -> N309;
  N100 -> N309;
  N116 -> N309;
  N132 -> N309;
  N148 -> N309;
  N164 -> N309;
}

[neubig]

Hmm, I was not able to reproduce in my environment unfortunately... Does this test case break for you? Can you make one that does? I also tried where I made one of the elements only have a single batch element.

BOOST_AUTO_TEST_CASE( concatenate_cols_big ) {
  dynet::ComputationGraph cg;
  vector<float> f(512*32,0.f);
  Expression in = input(cg, Dim({512}, 32), f);
  Expression y = concatenate_cols({in, in, in, in, in, in, in, in, in});
  Expression z = sum_batches(sum_elems(y));
  BOOST_CHECK(check_grad(mod, z, 0));
}

[pmichel31415]

OK so reinstalling after going back to 2.0 somehow solved it. I'm going to guess that was related with my installation maybe... anyway the problem has disappeared I'll close the issue