More documentations on Preparing training data for PAE, training GNN, and using it in Unity Demo

[cjerry1243]

Thank you for your great work.

I am able to train the PAE model, but it looks like the current pieces of code and resources cannot be connected to build a complete pipeline from data preparation, PAE training, GNN training, and assets importing for unity demo. (eg. the GNN training is missing. The data used for PAE training is different from the data preparation pipeline and unity demo. How to import the phase parameters to unity using deepphaseimporter?)

It would be helpful if more documentation could be provided.

[kotaxyz]

i agree it would be better if there is a video explaning the whole pipeline

[ShigemoriHakura]

And I tried to export the Quadruped data, but get this error.

(DeepPhaseSubmission) C:\Users\ShiroSaki\Desktop\DeepLearning\py\PyTorch\GNN>python Network.py
Loading Data/InputShape.txt  
Loading Data/OutputShape.txt  
Loading Data/InputNormalization.txt  
Loading Data/OutputNormalization.txt  
Warning: Number of gating features (168) and main features (1812) are not the same as input features (559).  
Traceback (most recent call last):  
  File "Network.py", line 163, in <module>  
    yPred, gPred = network(xBatch)  
  File "E:\anaconda3\envs\DeepPhaseSubmission\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl  
    return forward_call(*input, **kwargs)  
  File "Network.py", line 40, in forward  
    g = x[:, self.gating_indices]  
IndexError: index 1812 is out of bounds for dimension 0 with size 559

[cjerry1243]

Loading Data/InputShape.txt
Loading Data/OutputShape.txt
Loading Data/InputNormalization.txt
Loading Data/OutputNormalization.txt

Do you have those txt files? Network.py is the file used to train the neural controller, not used to export data.

[sebastianstarke]

Hi there - possibly I forgot to update the GNN network file for the quadruped dataset. The exported data should be right though - can you show me the generated InputLabels.txt file and I can have a quick look? The network model is used universally across different datasets and the indices need to be adjusted depending on the dataset.

[cjerry1243]

I have the same question on how to export the PAE phase to get the txt data for GNN training.

I am currently stuck here at deepphaseimporter. Looks like nothing was generated. Is it supposed to generate the data for GNN training?

[ShigemoriHakura]

Hi there - possibly I forgot to update the GNN network file for the quadruped dataset. The exported data should be right though - can you show me the generated InputLabels.txt file and I can have a quick look? The network model is used universally across different datasets and the indices need to be adjusted depending on the dataset.

The inputlabel I exported:

[0] TrajectoryPosition1X
[1] TrajectoryPosition1Z
[2] TrajectoryDirection1X
[3] TrajectoryDirection1Z
[4] TrajectoryVelocity1X
[5] TrajectoryVelocity1Z
[6] Actions1-1
[7] Actions1-2
[8] Actions1-3
[9] TrajectoryPosition2X
[10] TrajectoryPosition2Z
[11] TrajectoryDirection2X
[12] TrajectoryDirection2Z
[13] TrajectoryVelocity2X
[14] TrajectoryVelocity2Z
[15] Actions2-1
[16] Actions2-2
[17] Actions2-3
[18] TrajectoryPosition3X
[19] TrajectoryPosition3Z
[20] TrajectoryDirection3X
[21] TrajectoryDirection3Z
[22] TrajectoryVelocity3X
[23] TrajectoryVelocity3Z
[24] Actions3-1
[25] Actions3-2
[26] Actions3-3
[27] TrajectoryPosition4X
[28] TrajectoryPosition4Z
[29] TrajectoryDirection4X
[30] TrajectoryDirection4Z
[31] TrajectoryVelocity4X
[32] TrajectoryVelocity4Z
[33] Actions4-1
[34] Actions4-2
[35] Actions4-3
[36] TrajectoryPosition5X
[37] TrajectoryPosition5Z
[38] TrajectoryDirection5X
[39] TrajectoryDirection5Z
[40] TrajectoryVelocity5X
[41] TrajectoryVelocity5Z
[42] Actions5-1
[43] Actions5-2
[44] Actions5-3
[45] TrajectoryPosition6X
[46] TrajectoryPosition6Z
[47] TrajectoryDirection6X
[48] TrajectoryDirection6Z
[49] TrajectoryVelocity6X
[50] TrajectoryVelocity6Z
[51] Actions6-1
[52] Actions6-2
[53] Actions6-3
[54] TrajectoryPosition7X
[55] TrajectoryPosition7Z
[56] TrajectoryDirection7X
[57] TrajectoryDirection7Z
[58] TrajectoryVelocity7X
[59] TrajectoryVelocity7Z
[60] Actions7-1
[61] Actions7-2
[62] Actions7-3
[63] TrajectoryPosition8X
[64] TrajectoryPosition8Z
[65] TrajectoryDirection8X
[66] TrajectoryDirection8Z
[67] TrajectoryVelocity8X
[68] TrajectoryVelocity8Z
[69] Actions8-1
[70] Actions8-2
[71] Actions8-3
[72] TrajectoryPosition9X
[73] TrajectoryPosition9Z
[74] TrajectoryDirection9X
[75] TrajectoryDirection9Z
[76] TrajectoryVelocity9X
[77] TrajectoryVelocity9Z
[78] Actions9-1
[79] Actions9-2
[80] Actions9-3
[81] TrajectoryPosition10X
[82] TrajectoryPosition10Z
[83] TrajectoryDirection10X
[84] TrajectoryDirection10Z
[85] TrajectoryVelocity10X
[86] TrajectoryVelocity10Z
[87] Actions10-1
[88] Actions10-2
[89] Actions10-3
[90] TrajectoryPosition11X
[91] TrajectoryPosition11Z
[92] TrajectoryDirection11X
[93] TrajectoryDirection11Z
[94] TrajectoryVelocity11X
[95] TrajectoryVelocity11Z
[96] Actions11-1
[97] Actions11-2
[98] Actions11-3
[99] TrajectoryPosition12X
[100] TrajectoryPosition12Z
[101] TrajectoryDirection12X
[102] TrajectoryDirection12Z
[103] TrajectoryVelocity12X
[104] TrajectoryVelocity12Z
[105] Actions12-1
[106] Actions12-2
[107] Actions12-3
[108] TrajectoryPosition13X
[109] TrajectoryPosition13Z
[110] TrajectoryDirection13X
[111] TrajectoryDirection13Z
[112] TrajectoryVelocity13X
[113] TrajectoryVelocity13Z
[114] Actions13-1
[115] Actions13-2
[116] Actions13-3
[117] Bone1HipsPositionX
[118] Bone1HipsPositionY
[119] Bone1HipsPositionZ
[120] Bone1HipsForwardX
[121] Bone1HipsForwardY
[122] Bone1HipsForwardZ
[123] Bone1HipsUpX
[124] Bone1HipsUpY
[125] Bone1HipsUpZ
[126] Bone1HipsVelocityX
[127] Bone1HipsVelocityY
[128] Bone1HipsVelocityZ
[129] Bone2LeftUpLegPositionX
[130] Bone2LeftUpLegPositionY
[131] Bone2LeftUpLegPositionZ
[132] Bone2LeftUpLegForwardX
[133] Bone2LeftUpLegForwardY
[134] Bone2LeftUpLegForwardZ
[135] Bone2LeftUpLegUpX
[136] Bone2LeftUpLegUpY
[137] Bone2LeftUpLegUpZ
[138] Bone2LeftUpLegVelocityX
[139] Bone2LeftUpLegVelocityY
[140] Bone2LeftUpLegVelocityZ
[141] Bone3LeftLegPositionX
[142] Bone3LeftLegPositionY
[143] Bone3LeftLegPositionZ
[144] Bone3LeftLegForwardX
[145] Bone3LeftLegForwardY
[146] Bone3LeftLegForwardZ
[147] Bone3LeftLegUpX
[148] Bone3LeftLegUpY
[149] Bone3LeftLegUpZ
[150] Bone3LeftLegVelocityX
[151] Bone3LeftLegVelocityY
[152] Bone3LeftLegVelocityZ
[153] Bone4LeftFootPositionX
[154] Bone4LeftFootPositionY
[155] Bone4LeftFootPositionZ
[156] Bone4LeftFootForwardX
[157] Bone4LeftFootForwardY
[158] Bone4LeftFootForwardZ
[159] Bone4LeftFootUpX
[160] Bone4LeftFootUpY
[161] Bone4LeftFootUpZ
[162] Bone4LeftFootVelocityX
[163] Bone4LeftFootVelocityY
[164] Bone4LeftFootVelocityZ
[165] Bone5LeftFootSitePositionX
[166] Bone5LeftFootSitePositionY
[167] Bone5LeftFootSitePositionZ
[168] Bone5LeftFootSiteForwardX
[169] Bone5LeftFootSiteForwardY
[170] Bone5LeftFootSiteForwardZ
[171] Bone5LeftFootSiteUpX
[172] Bone5LeftFootSiteUpY
[173] Bone5LeftFootSiteUpZ
[174] Bone5LeftFootSiteVelocityX
[175] Bone5LeftFootSiteVelocityY
[176] Bone5LeftFootSiteVelocityZ
[177] Bone6RightUpLegPositionX
[178] Bone6RightUpLegPositionY
[179] Bone6RightUpLegPositionZ
[180] Bone6RightUpLegForwardX
[181] Bone6RightUpLegForwardY
[182] Bone6RightUpLegForwardZ
[183] Bone6RightUpLegUpX
[184] Bone6RightUpLegUpY
[185] Bone6RightUpLegUpZ
[186] Bone6RightUpLegVelocityX
[187] Bone6RightUpLegVelocityY
[188] Bone6RightUpLegVelocityZ
[189] Bone7RightLegPositionX
[190] Bone7RightLegPositionY
[191] Bone7RightLegPositionZ
[192] Bone7RightLegForwardX
[193] Bone7RightLegForwardY
[194] Bone7RightLegForwardZ
[195] Bone7RightLegUpX
[196] Bone7RightLegUpY
[197] Bone7RightLegUpZ
[198] Bone7RightLegVelocityX
[199] Bone7RightLegVelocityY
[200] Bone7RightLegVelocityZ
[201] Bone8RightFootPositionX
[202] Bone8RightFootPositionY
[203] Bone8RightFootPositionZ
[204] Bone8RightFootForwardX
[205] Bone8RightFootForwardY
[206] Bone8RightFootForwardZ
[207] Bone8RightFootUpX
[208] Bone8RightFootUpY
[209] Bone8RightFootUpZ
[210] Bone8RightFootVelocityX
[211] Bone8RightFootVelocityY
[212] Bone8RightFootVelocityZ
[213] Bone9RightFootSitePositionX
[214] Bone9RightFootSitePositionY
[215] Bone9RightFootSitePositionZ
[216] Bone9RightFootSiteForwardX
[217] Bone9RightFootSiteForwardY
[218] Bone9RightFootSiteForwardZ
[219] Bone9RightFootSiteUpX
[220] Bone9RightFootSiteUpY
[221] Bone9RightFootSiteUpZ
[222] Bone9RightFootSiteVelocityX
[223] Bone9RightFootSiteVelocityY
[224] Bone9RightFootSiteVelocityZ
[225] Bone10SpinePositionX
[226] Bone10SpinePositionY
[227] Bone10SpinePositionZ
[228] Bone10SpineForwardX
[229] Bone10SpineForwardY
[230] Bone10SpineForwardZ
[231] Bone10SpineUpX
[232] Bone10SpineUpY
[233] Bone10SpineUpZ
[234] Bone10SpineVelocityX
[235] Bone10SpineVelocityY
[236] Bone10SpineVelocityZ
[237] Bone11Spine1PositionX
[238] Bone11Spine1PositionY
[239] Bone11Spine1PositionZ
[240] Bone11Spine1ForwardX
[241] Bone11Spine1ForwardY
[242] Bone11Spine1ForwardZ
[243] Bone11Spine1UpX
[244] Bone11Spine1UpY
[245] Bone11Spine1UpZ
[246] Bone11Spine1VelocityX
[247] Bone11Spine1VelocityY
[248] Bone11Spine1VelocityZ
[249] Bone12LeftShoulderPositionX
[250] Bone12LeftShoulderPositionY
[251] Bone12LeftShoulderPositionZ
[252] Bone12LeftShoulderForwardX
[253] Bone12LeftShoulderForwardY
[254] Bone12LeftShoulderForwardZ
[255] Bone12LeftShoulderUpX
[256] Bone12LeftShoulderUpY
[257] Bone12LeftShoulderUpZ
[258] Bone12LeftShoulderVelocityX
[259] Bone12LeftShoulderVelocityY
[260] Bone12LeftShoulderVelocityZ
[261] Bone13LeftArmPositionX
[262] Bone13LeftArmPositionY
[263] Bone13LeftArmPositionZ
[264] Bone13LeftArmForwardX
[265] Bone13LeftArmForwardY
[266] Bone13LeftArmForwardZ
[267] Bone13LeftArmUpX
[268] Bone13LeftArmUpY
[269] Bone13LeftArmUpZ
[270] Bone13LeftArmVelocityX
[271] Bone13LeftArmVelocityY
[272] Bone13LeftArmVelocityZ
[273] Bone14LeftForeArmPositionX
[274] Bone14LeftForeArmPositionY
[275] Bone14LeftForeArmPositionZ
[276] Bone14LeftForeArmForwardX
[277] Bone14LeftForeArmForwardY
[278] Bone14LeftForeArmForwardZ
[279] Bone14LeftForeArmUpX
[280] Bone14LeftForeArmUpY
[281] Bone14LeftForeArmUpZ
[282] Bone14LeftForeArmVelocityX
[283] Bone14LeftForeArmVelocityY
[284] Bone14LeftForeArmVelocityZ
[285] Bone15LeftHandPositionX
[286] Bone15LeftHandPositionY
[287] Bone15LeftHandPositionZ
[288] Bone15LeftHandForwardX
[289] Bone15LeftHandForwardY
[290] Bone15LeftHandForwardZ
[291] Bone15LeftHandUpX
[292] Bone15LeftHandUpY
[293] Bone15LeftHandUpZ
[294] Bone15LeftHandVelocityX
[295] Bone15LeftHandVelocityY
[296] Bone15LeftHandVelocityZ
[297] Bone16LeftHandSitePositionX
[298] Bone16LeftHandSitePositionY
[299] Bone16LeftHandSitePositionZ
[300] Bone16LeftHandSiteForwardX
[301] Bone16LeftHandSiteForwardY
[302] Bone16LeftHandSiteForwardZ
[303] Bone16LeftHandSiteUpX
[304] Bone16LeftHandSiteUpY
[305] Bone16LeftHandSiteUpZ
[306] Bone16LeftHandSiteVelocityX
[307] Bone16LeftHandSiteVelocityY
[308] Bone16LeftHandSiteVelocityZ
[309] Bone17NeckPositionX
[310] Bone17NeckPositionY
[311] Bone17NeckPositionZ
[312] Bone17NeckForwardX
[313] Bone17NeckForwardY
[314] Bone17NeckForwardZ
[315] Bone17NeckUpX
[316] Bone17NeckUpY
[317] Bone17NeckUpZ
[318] Bone17NeckVelocityX
[319] Bone17NeckVelocityY
[320] Bone17NeckVelocityZ
[321] Bone18HeadPositionX
[322] Bone18HeadPositionY
[323] Bone18HeadPositionZ
[324] Bone18HeadForwardX
[325] Bone18HeadForwardY
[326] Bone18HeadForwardZ
[327] Bone18HeadUpX
[328] Bone18HeadUpY
[329] Bone18HeadUpZ
[330] Bone18HeadVelocityX
[331] Bone18HeadVelocityY
[332] Bone18HeadVelocityZ
[333] Bone19RightShoulderPositionX
[334] Bone19RightShoulderPositionY
[335] Bone19RightShoulderPositionZ
[336] Bone19RightShoulderForwardX
[337] Bone19RightShoulderForwardY
[338] Bone19RightShoulderForwardZ
[339] Bone19RightShoulderUpX
[340] Bone19RightShoulderUpY
[341] Bone19RightShoulderUpZ
[342] Bone19RightShoulderVelocityX
[343] Bone19RightShoulderVelocityY
[344] Bone19RightShoulderVelocityZ
[345] Bone20RightArmPositionX
[346] Bone20RightArmPositionY
[347] Bone20RightArmPositionZ
[348] Bone20RightArmForwardX
[349] Bone20RightArmForwardY
[350] Bone20RightArmForwardZ
[351] Bone20RightArmUpX
[352] Bone20RightArmUpY
[353] Bone20RightArmUpZ
[354] Bone20RightArmVelocityX
[355] Bone20RightArmVelocityY
[356] Bone20RightArmVelocityZ
[357] Bone21RightForeArmPositionX
[358] Bone21RightForeArmPositionY
[359] Bone21RightForeArmPositionZ
[360] Bone21RightForeArmForwardX
[361] Bone21RightForeArmForwardY
[362] Bone21RightForeArmForwardZ
[363] Bone21RightForeArmUpX
[364] Bone21RightForeArmUpY
[365] Bone21RightForeArmUpZ
[366] Bone21RightForeArmVelocityX
[367] Bone21RightForeArmVelocityY
[368] Bone21RightForeArmVelocityZ
[369] Bone22RightHandPositionX
[370] Bone22RightHandPositionY
[371] Bone22RightHandPositionZ
[372] Bone22RightHandForwardX
[373] Bone22RightHandForwardY
[374] Bone22RightHandForwardZ
[375] Bone22RightHandUpX
[376] Bone22RightHandUpY
[377] Bone22RightHandUpZ
[378] Bone22RightHandVelocityX
[379] Bone22RightHandVelocityY
[380] Bone22RightHandVelocityZ
[381] Bone23RightHandSitePositionX
[382] Bone23RightHandSitePositionY
[383] Bone23RightHandSitePositionZ
[384] Bone23RightHandSiteForwardX
[385] Bone23RightHandSiteForwardY
[386] Bone23RightHandSiteForwardZ
[387] Bone23RightHandSiteUpX
[388] Bone23RightHandSiteUpY
[389] Bone23RightHandSiteUpZ
[390] Bone23RightHandSiteVelocityX
[391] Bone23RightHandSiteVelocityY
[392] Bone23RightHandSiteVelocityZ
[393] Bone24TailPositionX
[394] Bone24TailPositionY
[395] Bone24TailPositionZ
[396] Bone24TailForwardX
[397] Bone24TailForwardY
[398] Bone24TailForwardZ
[399] Bone24TailUpX
[400] Bone24TailUpY
[401] Bone24TailUpZ
[402] Bone24TailVelocityX
[403] Bone24TailVelocityY
[404] Bone24TailVelocityZ
[405] Bone25Tail1PositionX
[406] Bone25Tail1PositionY
[407] Bone25Tail1PositionZ
[408] Bone25Tail1ForwardX
[409] Bone25Tail1ForwardY
[410] Bone25Tail1ForwardZ
[411] Bone25Tail1UpX
[412] Bone25Tail1UpY
[413] Bone25Tail1UpZ
[414] Bone25Tail1VelocityX
[415] Bone25Tail1VelocityY
[416] Bone25Tail1VelocityZ
[417] Bone26Tail1SitePositionX
[418] Bone26Tail1SitePositionY
[419] Bone26Tail1SitePositionZ
[420] Bone26Tail1SiteForwardX
[421] Bone26Tail1SiteForwardY
[422] Bone26Tail1SiteForwardZ
[423] Bone26Tail1SiteUpX
[424] Bone26Tail1SiteUpY
[425] Bone26Tail1SiteUpZ
[426] Bone26Tail1SiteVelocityX
[427] Bone26Tail1SiteVelocityY
[428] Bone26Tail1SiteVelocityZ
[429] PhaseSpace-1
[430] PhaseSpace-2
[431] PhaseSpace-3
[432] PhaseSpace-4
[433] PhaseSpace-5
[434] PhaseSpace-6
[435] PhaseSpace-7
[436] PhaseSpace-8
[437] PhaseSpace-9
[438] PhaseSpace-10
[439] PhaseSpace-11
[440] PhaseSpace-12
[441] PhaseSpace-13
[442] PhaseSpace-14
[443] PhaseSpace-15
[444] PhaseSpace-16
[445] PhaseSpace-17
[446] PhaseSpace-18
[447] PhaseSpace-19
[448] PhaseSpace-20
[449] PhaseSpace-21
[450] PhaseSpace-22
[451] PhaseSpace-23
[452] PhaseSpace-24
[453] PhaseSpace-25
[454] PhaseSpace-26
[455] PhaseSpace-27
[456] PhaseSpace-28
[457] PhaseSpace-29
[458] PhaseSpace-30
[459] PhaseSpace-31
[460] PhaseSpace-32
[461] PhaseSpace-33
[462] PhaseSpace-34
[463] PhaseSpace-35
[464] PhaseSpace-36
[465] PhaseSpace-37
[466] PhaseSpace-38
[467] PhaseSpace-39
[468] PhaseSpace-40
[469] PhaseSpace-41
[470] PhaseSpace-42
[471] PhaseSpace-43
[472] PhaseSpace-44
[473] PhaseSpace-45
[474] PhaseSpace-46
[475] PhaseSpace-47
[476] PhaseSpace-48
[477] PhaseSpace-49
[478] PhaseSpace-50
[479] PhaseSpace-51
[480] PhaseSpace-52
[481] PhaseSpace-53
[482] PhaseSpace-54
[483] PhaseSpace-55
[484] PhaseSpace-56
[485] PhaseSpace-57
[486] PhaseSpace-58
[487] PhaseSpace-59
[488] PhaseSpace-60
[489] PhaseSpace-61
[490] PhaseSpace-62
[491] PhaseSpace-63
[492] PhaseSpace-64
[493] PhaseSpace-65
[494] PhaseSpace-66
[495] PhaseSpace-67
[496] PhaseSpace-68
[497] PhaseSpace-69
[498] PhaseSpace-70
[499] PhaseSpace-71
[500] PhaseSpace-72
[501] PhaseSpace-73
[502] PhaseSpace-74
[503] PhaseSpace-75
[504] PhaseSpace-76
[505] PhaseSpace-77
[506] PhaseSpace-78
[507] PhaseSpace-79
[508] PhaseSpace-80
[509] PhaseSpace-81
[510] PhaseSpace-82
[511] PhaseSpace-83
[512] PhaseSpace-84
[513] PhaseSpace-85
[514] PhaseSpace-86
[515] PhaseSpace-87
[516] PhaseSpace-88
[517] PhaseSpace-89
[518] PhaseSpace-90
[519] PhaseSpace-91
[520] PhaseSpace-92
[521] PhaseSpace-93
[522] PhaseSpace-94
[523] PhaseSpace-95
[524] PhaseSpace-96
[525] PhaseSpace-97
[526] PhaseSpace-98
[527] PhaseSpace-99
[528] PhaseSpace-100
[529] PhaseSpace-101
[530] PhaseSpace-102
[531] PhaseSpace-103
[532] PhaseSpace-104
[533] PhaseSpace-105
[534] PhaseSpace-106
[535] PhaseSpace-107
[536] PhaseSpace-108
[537] PhaseSpace-109
[538] PhaseSpace-110
[539] PhaseSpace-111
[540] PhaseSpace-112
[541] PhaseSpace-113
[542] PhaseSpace-114
[543] PhaseSpace-115
[544] PhaseSpace-116
[545] PhaseSpace-117
[546] PhaseSpace-118
[547] PhaseSpace-119
[548] PhaseSpace-120
[549] PhaseSpace-121
[550] PhaseSpace-122
[551] PhaseSpace-123
[552] PhaseSpace-124
[553] PhaseSpace-125
[554] PhaseSpace-126
[555] PhaseSpace-127
[556] PhaseSpace-128
[557] PhaseSpace-129
[558] PhaseSpace-130

[sebastianstarke]

I see, thanks for sharing. Yes in this case the GNN input indices need to be adjusted to:

gating_indices = torch.tensor([(429+ i) for i in range(130)]) main_indices = torch.tensor([(0 + i) for i in range(429)])

This is to take all features apart from phase as input to the experts and only the phases as input to the gating network.

[happyday521]

@sebastianstarke @ShigemoriHakura Can you tell me how to export the txt files(e.g., InputLabels.txt, OutputLables.txt, etc.) for the quadruped dataset (or my own MoCap data) from the Unity, in order to train the GNN network (instead of PAE)?

Thanks!

[sebastianstarke]

@happyday521 To export training data for the GNN motion controller the following needs to be done:

1) AI4Animation -> Tools -> AssetPipeline 2) Drag & Drop the "QuadrupedPipeline" asset from the quadruped demo folder into the "Setup" field. 3) Click "Refresh" which should display all assets currently loaded in the MotionEditor. 4) Click "Process"

Let me know if that works, thanks.

[ShigemoriHakura]

@sebastianstarke @ShigemoriHakura Can you tell me how to export the txt files(e.g., InputLabels.txt, OutputLables.txt, etc.) for the quadruped dataset (or my own MoCap data) from the Unity, in order to train the GNN network (instead of PAE)?

Thanks!

Need to fix the process progress due to missing StandardValues and MirroredValues in StyleModule of every bvh asset. I modified the function starts with line 319 in StyleModule.cs to recompute the StandardValues and MirroredValues

public float GetValue(Frame frame, bool mirrored) {

    bool needRe = false;

    if (StandardValues == null || MirroredValues == null || StandardValues.Length == 0 || MirroredValues.Length == 0)
    {
        needRe = true;
    }

     if (needRe)
     {
        StandardValues = new float[Module.Asset.Frames.Length];
        StandardValues.SetAll(0f);
        MirroredValues = new float[Module.Asset.Frames.Length];
        MirroredValues.SetAll(0f);
        for (int i = 0; i < Module.Asset.Frames.Length; i++)
        {
            Compute(Module.Asset.GetFrame(i));
        }
        Module.Asset.Save();
        Debug.Log("Recomputed " + Module.Asset.name);
    }

    return mirrored ? MirroredValues[frame.Index-1] : StandardValues[frame.Index-1];
}

[happyday521]

@happyday521 To export training data for the GNN motion controller the following needs to be done:

1. AI4Animation -> Tools -> AssetPipeline
2. Drag & Drop the "QuadrupedPipeline" asset from the quadruped demo folder into the "Setup" field.
3. Click "Refresh" which should display all assets currently loaded in the MotionEditor.
4. Click "Process"

Let me know if that works, thanks.

Hi. I open the "MotionCapture.unity" file in the quadruped demo folder and do as you said. Note I set the "Mode" to "Export Controller".

However, when I click "Process" , I got the error as follows:

Can you help me? Thanks very much!

[happyday521]

@sebastianstarke @ShigemoriHakura Can you tell me how to export the txt files(e.g., InputLabels.txt, OutputLables.txt, etc.) for the quadruped dataset (or my own MoCap data) from the Unity, in order to train the GNN network (instead of PAE)? Thanks!

Need to fix the process progress due to missing StandardValues and MirroredValues in StyleModule of every bvh asset. I modified the function starts with line 319 in StyleModule.cs to recompute the StandardValues and MirroredValues

public float GetValue(Frame frame, bool mirrored) {

    bool needRe = false;

    if (StandardValues == null || MirroredValues == null || StandardValues.Length == 0 || MirroredValues.Length == 0)
    {
        needRe = true;
    }

     if (needRe)
     {
        StandardValues = new float[Module.Asset.Frames.Length];
        StandardValues.SetAll(0f);
        MirroredValues = new float[Module.Asset.Frames.Length];
        MirroredValues.SetAll(0f);
        for (int i = 0; i < Module.Asset.Frames.Length; i++)
        {
            Compute(Module.Asset.GetFrame(i));
        }
        Module.Asset.Save();
        Debug.Log("Recomputed " + Module.Asset.name);
    }

    return mirrored ? MirroredValues[frame.Index-1] : StandardValues[frame.Index-1];
}

Thanks! Need I set the "Mode" to "Export Controller" when using "QuadrupedPipeline" to export data?

[ShigemoriHakura]

@sebastianstarke @ShigemoriHakura Can you tell me how to export the txt files(e.g., InputLabels.txt, OutputLables.txt, etc.) for the quadruped dataset (or my own MoCap data) from the Unity, in order to train the GNN network (instead of PAE)? Thanks!

Need to fix the process progress due to missing StandardValues and MirroredValues in StyleModule of every bvh asset. I modified the function starts with line 319 in StyleModule.cs to recompute the StandardValues and MirroredValues

public float GetValue(Frame frame, bool mirrored) {

    bool needRe = false;

    if (StandardValues == null || MirroredValues == null || StandardValues.Length == 0 || MirroredValues.Length == 0)
    {
        needRe = true;
    }

     if (needRe)
     {
        StandardValues = new float[Module.Asset.Frames.Length];
        StandardValues.SetAll(0f);
        MirroredValues = new float[Module.Asset.Frames.Length];
        MirroredValues.SetAll(0f);
        for (int i = 0; i < Module.Asset.Frames.Length; i++)
        {
            Compute(Module.Asset.GetFrame(i));
        }
        Module.Asset.Save();
        Debug.Log("Recomputed " + Module.Asset.name);
    }

    return mirrored ? MirroredValues[frame.Index-1] : StandardValues[frame.Index-1];
}

Thanks! Need I set the "Mode" to "Export Controller" when using "QuadrupedPipeline" to export data?

Should process assets first then export I think

[cjerry1243]

@ShigemoriHakura @sebastianstarke Thank you, the pipeline works.

[cjerry1243]

@ShigemoriHakura @sebastianstarke I have a couple of questions regarding the Sig22 quadruped pipeline and demo.

1. What are the user inputs for controlling the quadruped in the game? I tried every key but none works.
2. How are the phases extracted during the quadruped "Export Controller" pipeline? Shouldn't the phases come from the trained PAE encoder? If yes, where is that model located?
3. The foreleg of the quadruped is a little bit weird after I followed the instruction you posted here and dragged in the exported onnx as the Neural Network Model. https://user-images.githubusercontent.com/26675834/218866744-0f23772c-09c3-4f71-b64a-3af900a58f4a.mp4

I appreciate your time for the reply.

[paulstarke]

Hi everyone, thanks for letting us know about the issues. Due to the high frequency of questions about the Sig22 pipeline I made a quick step by step tutorial: https://youtu.be/3ASGrxNDd0k

Please use the current version of this repository as we made some changes in the past days and let us know if you still encounter issues.

@cjerry1243

1. The controller in this demo only processes inputs from a gamepad.
2. A trained PAE model for the quadruped is currently not in the repo.
3. This seems to come from the fact that the future trajectory is moving which acts as a control signal for GNN network.

[cjerry1243]

Much appreciated 👍 Thank you for your time doing this tutorial.