Closed ai-honzik closed 4 years ago
This may be related to cuda's printf function and the buffer size: https://stackoverflow.com/questions/15421626/incomplete-output-from-printf-called-on-device https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#formatted-output
If needed, I can post vxa and vxd files that we used.
Thanks, please. I'll try to reproduce the error and see what's going on.
vxa
<VXA Version="1.1">
<Simulator>
<Integration>
<DtFrac>0.5</DtFrac>
</Integration>
<Damping>
<BondDampingZ>0.1</BondDampingZ>
<ColDampingZ>1.0</ColDampingZ>
<SlowDampingZ>1.0</SlowDampingZ>
</Damping>
<StopCondition>
<StopConditionFormula>
<mtSUB>
<mtVAR>t</mtVAR>
<mtCONST>3</mtCONST>
</mtSUB>
</StopConditionFormula>
</StopCondition>
<RecordHistory>
<RecordStepSize>100</RecordStepSize>
<RecordVoxel>1</RecordVoxel>
<RecordLink>0</RecordLink>
</RecordHistory>
<AttachDetach>
<EnableCollision>1</EnableCollision>
<SafetyGuard>500</SafetyGuard>
</AttachDetach>
<EnableSignals>0</EnableSignals>
<EnableCilia>0</EnableCilia>
<SavePositionOfAllVoxels>0</SavePositionOfAllVoxels>
<MaxDistInVoxelLengthsToCountAsPair>0</MaxDistInVoxelLengthsToCountAsPair>
</Simulator>
<Environment>
<Gravity>
<GravEnabled>1</GravEnabled>
<GravAcc>-9.81</GravAcc>
<FloorEnabled>1</FloorEnabled>
</Gravity>
<Thermal>
<TempEnabled>1</TempEnabled>
<VaryTempEnabled>1</VaryTempEnabled>
<TempBase>25</TempBase>
<TempAmplitude>14.4714</TempAmplitude>
<TempPeriod>0.2</TempPeriod>
</Thermal>
</Environment>
<VXC Version="0.94">
<Lattice>
<Lattice_Dim>0.01</Lattice_Dim>
</Lattice>
<Palette>
<Material ID="1">
<Name>Material 0</Name>
<Display>
<Red>0.14099259409378007</Red>
<Green>0.08854556775050326</Green>
<Blue>0.809663997916719</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isTarget>0</isTarget>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Sticky>0</Sticky>
<Cilia>False</Cilia>
<PaceMakerPeriod>0</PaceMakerPeriod>
<signalValueDecay>0.9</signalValueDecay>
<signalTimeDelay>0.03</signalTimeDelay>
<inactivePeriod>0.03</inactivePeriod>
<MatModel>0</MatModel>
<Elastic_Mod>106570.82081835589</Elastic_Mod>
<Fail_Stress>0</Fail_Stress>
<Density>70271.85283485209</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.005021470540585884</CTE>
<uStatic>0.562913457012264</uStatic>
<uDynamic>0.13390041150158422</uDynamic>
<Cilia>0</Cilia>
</Mechanical>
</Material>
<Material ID="2">
<Name>Material 1</Name>
<Display>
<Red>0.08897459328396962</Red>
<Green>0.3376041933422086</Green>
<Blue>0.7445836931542802</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isTarget>0</isTarget>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Sticky>0</Sticky>
<Cilia>False</Cilia>
<PaceMakerPeriod>0</PaceMakerPeriod>
<signalValueDecay>0.9</signalValueDecay>
<signalTimeDelay>0.03</signalTimeDelay>
<inactivePeriod>0.03</inactivePeriod>
<MatModel>0</MatModel>
<Elastic_Mod>3697522.93419921</Elastic_Mod>
<Fail_Stress>0</Fail_Stress>
<Density>25822.4155534153</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.0009383215475447271</CTE>
<uStatic>0.45283506435743615</uStatic>
<uDynamic>0.11926600140511912</uDynamic>
<Cilia>0</Cilia>
</Mechanical>
</Material>
<Material ID="3">
<Name>Material 2</Name>
<Display>
<Red>0.4601874929672084</Red>
<Green>0.5085382660662086</Green>
<Blue>0.7814182148645693</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isTarget>0</isTarget>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Sticky>0</Sticky>
<Cilia>False</Cilia>
<PaceMakerPeriod>0</PaceMakerPeriod>
<signalValueDecay>0.9</signalValueDecay>
<signalTimeDelay>0.03</signalTimeDelay>
<inactivePeriod>0.03</inactivePeriod>
<MatModel>0</MatModel>
<Elastic_Mod>231875.7663659463</Elastic_Mod>
<Fail_Stress>0</Fail_Stress>
<Density>43219.452091459534</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.0034710450281608706</CTE>
<uStatic>0.20257894239116914</uStatic>
<uDynamic>0.3906711979002612</uDynamic>
<Cilia>0</Cilia>
</Mechanical>
</Material>
</Palette>
</VXC>
</VXA>
vxd:
<VXD>
<Structure replace="VXA.VXC.Structure" Compression="ASCII_READABLE">
<X_Voxels>20</X_Voxels>
<Y_Voxels>20</Y_Voxels>
<Z_Voxels>20</Z_Voxels>
<Data>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002200000000000000000000000000000000000000110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003300000000000000000011000000000000000000330000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
<Layer><![CDATA[0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]]></Layer>
</Data>
</Structure>
</VXD>
I think an easy fix would be to increase the buffer size, @skriegman does that in his branch (at least I think it's his). VX3_SimulationManager.cu:144
As I am not familiar with cuda programming, I cannot tell what is exactly wrong. The default buffer size on google colab (free version) is ~1MB afaik (yet it's not enough to create 100kB files?).
@ai-honzik to start let's simplify your genotype.
<VXA Version="1.1">
<Simulator>
<Integration>
<DtFrac>0.9</DtFrac><!--was 0.5-->
</Integration>
<StopCondition>
<StopConditionFormula>
<mtSUB>
<mtVAR>t</mtVAR>
<mtCONST>3</mtCONST>
</mtSUB>
</StopConditionFormula>
</StopCondition>
<RecordHistory>
<RecordStepSize>100</RecordStepSize>
<RecordVoxel>1</RecordVoxel>
<RecordLink>0</RecordLink>
</RecordHistory>
</Simulator>
<VXC Version="0.94">
<Lattice>
<Lattice_Dim>0.01</Lattice_Dim>
</Lattice>
<Palette>
<Material ID="1">
<Name>Material 0</Name>
<Display>
<Red>0.14099259409378007</Red>
<Green>0.08854556775050326</Green>
<Blue>0.809663997916719</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Elastic_Mod>106570.82081835589</Elastic_Mod>
<Density>70271.85283485209</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.005021470540585884</CTE>
<uStatic>0.562913457012264</uStatic>
<uDynamic>0.13390041150158422</uDynamic>
</Mechanical>
</Material>
<Material ID="2">
<Name>Material 1</Name>
<Display>
<Red>0.08897459328396962</Red>
<Green>0.3376041933422086</Green>
<Blue>0.7445836931542802</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Elastic_Mod>3697522.93419921</Elastic_Mod>
<Density>25822.4155534153</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.0009383215475447271</CTE>
<uStatic>0.45283506435743615</uStatic>
<uDynamic>0.11926600140511912</uDynamic>
</Mechanical>
</Material>
<Material ID="3">
<Name>Material 2</Name>
<Display>
<Red>0.4601874929672084</Red>
<Green>0.5085382660662086</Green>
<Blue>0.7814182148645693</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Elastic_Mod>231875.7663659463</Elastic_Mod>
<Density>43219.452091459534</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.0034710450281608706</CTE>
<uStatic>0.20257894239116914</uStatic>
<uDynamic>0.3906711979002612</uDynamic>
</Mechanical>
</Material>
</Palette>
</VXC>
</VXA>
@ai-honzik I'm curious, why did you adjust the damping?
@skriegman we have some default presets, to be honest I thought these are default settings (https://github.com/ai-honzik/sr2020/#simulatorpy) of voxcraft-sim as well (but looking at sim code I guess I was wrong...)
Customized evosorocore - https://github.com/ai-honzik/sr2020/tree/master/evosorocore2
The time step calculation is usually very good. You likely won't need dtFrac to be so low (i.e. <0.9) if you use the default bond damping.
I would delete the
Actually we get a lot of Divergence errors, so I'll be trying to change DtFrac dynamically
oh and pacemakers. you're not using them so i am going to remove that from your vxa above too
you should not be getting any divergence errors!!
also i like how precise your material colors are :smile: Is that the actual color of your robot?
also i like how precise your material colors are Is that the actual color of your robot?
They are :D but they get assigned randomly so we could distinguish materials
I'll try to rerun vxa with your suggested edits, will let you know in a few minutes
Unfortunately, this did not solve the incomplete output.
@ai-honzik to start let's simplify your genotype.
in the vxa above I've put back the thermal section, we're using it to get some movement
for example, for this simulation, I get divergence error: vxa
<VXA Version="1.1">
<Simulator>
<Integration>
<DtFrac>0.5</DtFrac>
</Integration>
<StopCondition>
<StopConditionFormula>
<mtSUB>
<mtVAR>t</mtVAR>
<mtCONST>3</mtCONST>
</mtSUB>
</StopConditionFormula>
</StopCondition>
<RecordHistory>
<RecordStepSize>100</RecordStepSize>
<RecordVoxel>1</RecordVoxel>
<RecordLink>0</RecordLink>
</RecordHistory>
</Simulator>
<Environment>
<Thermal>
<TempEnabled>1</TempEnabled>
<VaryTempEnabled>1</VaryTempEnabled>
<TempBase>25</TempBase>
<TempAmplitude>14.4714</TempAmplitude>
<TempPeriod>0.2</TempPeriod>
</Thermal>
</Environment>
<VXC Version="0.94">
<Lattice>
<Lattice_Dim>0.01</Lattice_Dim>
</Lattice>
<Palette>
<Material ID="1">
<Name>Material 0</Name>
<Display>
<Red>0.9216396909361838</Red>
<Green>0.20588867626862484</Green>
<Blue>0.6966622628078402</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Elastic_Mod>946491.1360389724</Elastic_Mod>
<Density>551.4193492855047</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.095190450766868</CTE>
<uStatic>0.1310028377739153</uStatic>
<uDynamic>0.48415081021552575</uDynamic>
</Mechanical>
</Material>
<Material ID="2">
<Name>Material 1</Name>
<Display>
<Red>0.1613149961373923</Red>
<Green>0.4597767534821319</Green>
<Blue>0.635056433951152</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Elastic_Mod>9888786.548511231</Elastic_Mod>
<Density>1288.784230393132</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>0.0</CTE>
<uStatic>1.7313914907732975</uStatic>
<uDynamic>0.3596411213451568</uDynamic>
</Mechanical>
</Material>
<Material ID="3">
<Name>Material 2</Name>
<Display>
<Red>0.19957262653633367</Red>
<Green>0.4168893993758842</Green>
<Blue>0.4909096235059206</Blue>
<Alpha>1</Alpha>
</Display>
<Mechanical>
<isMeasured>1</isMeasured>
<Fixed>0</Fixed>
<Elastic_Mod>247681.03781932694</Elastic_Mod>
<Density>858.9800212477304</Density>
<Poissons_Ratio>0</Poissons_Ratio>
<CTE>-0.032648689989633654</CTE>
<uStatic>4.584110554187696</uStatic>
<uDynamic>0.3615878995260655</uDynamic>
</Mechanical>
</Material>
</Palette>
</VXC>
</VXA>
vxd
<VXD>
<Structure replace="VXA.VXC.Structure" Compression="ASCII_READABLE">
<X_Voxels>5</X_Voxels>
<Y_Voxels>1</Y_Voxels>
<Z_Voxels>1</Z_Voxels>
<Data>
<Layer><![CDATA[11123]]></Layer>
</Data>
</Structure>
</VXD>
Edit: I've got rid of some material properties, still getting divergence error.
@ai-honzik please try debugging a bit. For example, do you have the same problem with lower elastic mod?
@skriegman
With lower elastic mod I'm still getting divergence error, the behavior gets pretty weird once the values are under 1e5 (almost as if the ground did not exist). Increasing density had the largest impact, looks like that's what's making the simulation diverge so often.
For the history file output - is there a problem on our side?
@ai-honzik so there is a range of densities that are stable (and the history file is recorded) but another range of density values in which your simulation is not stable (and the history file is fine)?
@skriegman will take a look whether it has any impact on history files on colab (haven't checked that, was just checking for divergence errors).
And yes, for some densities the simulation seems to be exploding even with pretty low DtFrac.
Got it. It seems that I have polluted / complicated this thread. So: let's open up 2 new separate issues. The first for exploding sim at various parameter values. And a second issue (if it exists) of a stable sim leading to an invalid history file. For each, include the smallest example that demonstrates the issue. It is easier for us to debug 10 voxels than 10000 voxels, and more generally smaller vxd files. Thanks.
Sometimes, history files seem to be incomplete. This happens mostly on google colab but also on some of our personal computers.
Header seems to be missing and history file starts at 2.3 seconds (for 3s experiment). Most of the files have a similar problem.
If needed, I can post vxa and vxd files that we used.
e.g.