search

aws-neuron / aws-neuron-sdk

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services
https://aws.amazon.com/machine-learning/neuron/
Other
440 stars 145 forks source link

Compilation failure - Internal tensorizer error: NeuronValueNumbering:insertAtEnd() #947

Open ariveram2111 opened 1 month ago

ariveram2111 commented 1 month ago

Environment Python: 3.10.12 device : trn1.2xlarge neuronx-cc: 2.14.227.0+2d4f85be torch-neuronx: 2.1.2.2.2.0 neuronx-distributed: 0.8.0

$ dpkg-query -W -f='${binary:Package} ${Version}\n' | grep '^aws-neuron'
aws-neuronx-collectives 2.21.46.0-69b77134b
aws-neuronx-runtime-lib 2.21.41.0-fb1705f5f
aws-neuronx-tools 2.18.3.0
$ neuronx-cc --version
NeuronX Compiler version 2.14.227.0+2d4f85be

Python version 3.10.12
HWM version 2.14.0.227+2d4f85be
NumPy version 1.25.2

Failure logs using parallel compile utility

2024-08-15 14:01:20.000062:  1320  ERROR ||NEURON_CC_WRAPPER||: Failed compilation with ['neuronx-cc', 'compile', '--target=trn1', '--framework=XLA', '/tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_
12338015013531573548+00875417.hlo_module.pb', '--output', '/tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015013531573548+00875417.neff', '--auto-cast', 'none', '--verbose=35']: 2024-08-15T14:0
1:19Z [TEN404] (aten__add_add.31237) Internal tensorizer error: NeuronValueNumbering:insertAtEnd(): incompatible function arguments. The following argument types are supported: - Please open a support ticket at https://github.com/aws-neuron/aws-neuron-sdk/issues/new. You may also be able to obtain more information using the 'XLA_IR_DEBUG' and 'XLA_HLO_DEBUG' environment variables.

Compilation logs

$ XLA_IR_DEBUG=1 XLA_HLO_DEBUG=1 neuronx-cc compile --target=trn1 --framework=XLA /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015013531573548+00875417.hlo_module.pb
--output /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015013531573548+00875417.neff --auto-cast none --verbose=35
.....neuronxcc/starfish/penguin/ir/IRBuilder.py
neuronxcc/starfish/penguin/ir
neuronxcc/starfish/penguin
neuronxcc/starfish

[TEN404] (aten__add_add.31237) Internal tensorizer error: NeuronValueNumbering:insertAtEnd(): incompatible function arguments. The following argument types are supported: - Please open a support ticket at https://github.com/aws-neuron/aws-
neuron-sdk/issues/new. You may also be able to obtain more information using the 'XLA_IR_DEBUG' and 'XLA_HLO_DEBUG' environment variables.
2024-08-15T13:55:38Z INFO 1332 [root]: /usr/local/bin/neuronx-cc compile --target=trn1 --framework=XLA /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015013531573548+00875417.hlo_module.pb --ou
tput /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015013531573548+00875417.neff --auto-cast none --verbose=35
2024-08-15T13:55:38Z INFO 1454 [root]: XLA detected
2024-08-15T13:55:38Z INFO 1454 [root]: Pipeline: Frontend HHChecker WalrusDriver BIRLinker Kelper
2024-08-15T13:55:38Z INFO 1454 [root]: Intermediate files stored in /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/neuronxcc-zkczg6a7, output in /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f
45fb1498
2024-08-15T13:55:38Z INFO 1454 [pipeline.Pipeline.0]: Job Pipeline len(in_states) 1
2024-08-15T13:55:38Z INFO 1454 [pipeline.Pipeline.0]: Processing input #0
2024-08-15T13:55:38Z INFO 1454 [pipeline.Pipeline.0]: Running pipeline Pipeline.0
2024-08-15T13:55:38Z INFO 1454 [pipeline.Pipeline.0]: Starting job job.Frontend.0
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: Job Frontend len(in_states) 1
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: Processing input #0
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: Start model loading
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: IR signature: 3785806db282d2958935b153fceb45b5baa37c7735e2f76165e2114961ab6146 for model.MODULE_12338015013531573548+00875417.hlo_module.pb
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: Executing: /usr/local/lib/python3.10/site-packages/neuronxcc/starfish/bin/hlo2penguin --input /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/model.MODULE_12338015
013531573548+00875417.hlo_module.pb --out-dir ./ --output penguin.py --layers-per-module=1 --coalesce-all-gathers=false --coalesce-reduce-scatters=false --coalesce-all-reduces=false --emit-tensor-level-dropout-ops --emit-tensor-level-rng-o
ps
2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: DEBUG: needsModular? No. macCnt 1060634032128
INFO: Switching to single-module compile. PrePartitionPipe skipped.
INFO: Found compute bound graph
Replaced 90 dropout sequences with OffloadedDropout
INFO: HloMacCount has found 1060634032128
INFO: Traffic has found 1337459171
INFO: AIF 1586.04
HLO Ops used in computation: add batch-norm-grad batch-norm-training broadcast compare concatenate constant convert custom-call divide dot exponential gather get-tuple-element iota log multiply negate pad parameter reduce reshape scatter s
elect slice sqrt subtract transpose tuple
Invoking RemoveOptimizationBarriers pass

2024-08-15T13:55:38Z INFO 1454 [job.Frontend.0]: Start tensorization
2024-08-15T13:55:39Z WARNING 1454 [job.Frontend.0]: TVM not detected.
2024-08-15T13:55:39Z INFO 1454 [job.Frontend.0]: Num parallel jobs: 1
2024-08-15T13:55:39Z USER 1454 [root/Tensorizer/Tensorizer]: Running Tensorizer
2024-08-15T13:55:39Z INFO 1454 [Tensorizer]: Frontend found a single CU. Switching to flat flow.
2024-08-15T13:55:39Z INFO 1454 [Tensorizer]: Building model from Penguin script "penguin.py"...
2024-08-15T13:55:42Z INFO 1454 [Tensorizer]: Tensorizer options: --disable-bitcasted-transpose --dont-verify-after-all --fp32-cast=none --mm-transpose-type=fp32 --disable-expensive-checks --disable-max-stride-tiling --enable-replication --
max-local-tensor-tile-size-in-bytes=32768 --tensor-layout-p-order=0 --tensor-layout-b-order=1 --enable-advanced-delinearization --weight-coalescing-threshold=512 --enable-bir-converter=enable --sunda-batchnorm --enable-tritium-loopfusion -
-keep-remat-dma-transpose --enable-softmax-kernel
2024-08-15T13:55:42Z INFO 1454 [Tensorizer]: Building model from Penguin script "penguin.py"...
2024-08-15T13:55:46Z INFO 1454 [Tensorizer]: Successfully built model.
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/DoNothing]: Running DoNothing
2024-08-15T13:55:46Z INFO 1454 [DoNothing]: Finished (changed=True)
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/DoNothing]: DoNothing finished after 0.000 seconds
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/AliasDependencyInduction]: Running AliasDependencyInduction
2024-08-15T13:55:46Z INFO 1454 [AliasDependencyInduction]: Finished (changed=True)
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/AliasDependencyInduction]: AliasDependencyInduction finished after 0.045 seconds
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/TransformConvOp]: Running TransformConvOp
2024-08-15T13:55:46Z INFO 1454 [TransformConvOp]: Finished (changed=False)
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/TransformConvOp]: TransformConvOp finished after 0.164 seconds
2024-08-15T13:55:46Z INFO 1454 [sg0000/Tensorizer/LowerTensorOp]: Running LowerTensorOp
2024-08-15T13:55:48Z INFO 1454 [LowerTensorOp]: Finished (changed=True)
2024-08-15T13:55:48Z INFO 1454 [sg0000/Tensorizer/LowerTensorOp]: LowerTensorOp finished after 2.148 seconds
2024-08-15T13:55:48Z INFO 1454 [sg0000/Tensorizer/TensorOpSimplifier]: Running TensorOpSimplifier
2024-08-15T13:55:49Z INFO 1454 [TensorOpSimplifier]: Finished (changed=True)
2024-08-15T13:55:49Z INFO 1454 [sg0000/Tensorizer/TensorOpSimplifier]: TensorOpSimplifier finished after 0.685 seconds
2024-08-15T13:55:49Z INFO 1454 [sg0000/Tensorizer/CanonicalizeIR]: Running CanonicalizeIR
2024-08-15T13:55:49Z INFO 1454 [CanonicalizeIR]: Finished (changed=True)
2024-08-15T13:55:49Z INFO 1454 [sg0000/Tensorizer/CanonicalizeIR]: CanonicalizeIR finished after 0.465 seconds
2024-08-15T13:55:49Z INFO 1454 [sg0000/Tensorizer/LegalizeCCOpLayout]: Running LegalizeCCOpLayout
2024-08-15T13:55:50Z INFO 1454 [LegalizeCCOpLayout]: Finished (changed=False)
2024-08-15T13:55:50Z INFO 1454 [sg0000/Tensorizer/LegalizeCCOpLayout]: LegalizeCCOpLayout finished after 0.557 seconds
2024-08-15T13:55:50Z INFO 1454 [sg0000/Tensorizer/ResolveComplicatePredicates]: Running ResolveComplicatePredicates
2024-08-15T13:55:50Z INFO 1454 [ResolveComplicatePredicates]: Finished (changed=False)
2024-08-15T13:55:51Z INFO 1454 [sg0000/Tensorizer/ResolveComplicatePredicates]: ResolveComplicatePredicates finished after 0.449 seconds
2024-08-15T13:55:51Z INFO 1454 [sg0000/Tensorizer/AffinePredicateResolution]: Running AffinePredicateResolution
2024-08-15T13:55:51Z INFO 1454 [AffinePredicateResolution]: Finished (changed=False)
2024-08-15T13:55:51Z INFO 1454 [sg0000/Tensorizer/AffinePredicateResolution]: AffinePredicateResolution finished after 0.538 seconds
2024-08-15T13:55:51Z INFO 1454 [sg0000/Tensorizer/EliminateDivs]: Running EliminateDivs
2024-08-15T13:55:52Z INFO 1454 [EliminateDivs]: Finished (changed=False)
2024-08-15T13:55:52Z INFO 1454 [sg0000/Tensorizer/EliminateDivs]: EliminateDivs finished after 0.471 seconds
2024-08-15T13:55:52Z INFO 1454 [sg0000/Tensorizer/PerfectLoopNest]: Running PerfectLoopNest
2024-08-15T13:55:52Z INFO 1454 [PerfectLoopNest]: Finished (changed=False)
2024-08-15T13:55:52Z INFO 1454 [sg0000/Tensorizer/PerfectLoopNest]: PerfectLoopNest finished after 0.516 seconds
2024-08-15T13:55:52Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:55:56Z INFO 1454 [Simplifier]: Finished (changed=True)
2024-08-15T13:55:56Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 3.979 seconds
2024-08-15T13:55:56Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: Running GenericAccessSimplifier
2024-08-15T13:55:57Z INFO 1454 [GenericAccessSimplifier]: Finished (changed=False)
2024-08-15T13:55:57Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: GenericAccessSimplifier finished after 0.503 seconds
2024-08-15T13:55:57Z INFO 1454 [sg0000/Tensorizer/TCTransform]: Running TCTransform
2024-08-15T13:55:57Z INFO 1454 [TCTransform]: Finished (changed=True)
2024-08-15T13:55:57Z INFO 1454 [sg0000/Tensorizer/TCTransform]: TCTransform finished after 0.691 seconds
2024-08-15T13:55:57Z INFO 1454 [sg0000/Tensorizer/CommuteConcat]: Running CommuteConcat
2024-08-15T13:55:58Z INFO 1454 [CommuteConcat]: Finished (changed=False)
2024-08-15T13:55:58Z INFO 1454 [sg0000/Tensorizer/CommuteConcat]: CommuteConcat finished after 0.506 seconds
2024-08-15T13:55:58Z INFO 1454 [sg0000/Tensorizer/ExpandBatchNorm]: Running ExpandBatchNorm
2024-08-15T13:55:58Z INFO 1454 [ExpandBatchNorm]: Finished (changed=False)
2024-08-15T13:55:58Z INFO 1454 [sg0000/Tensorizer/ExpandBatchNorm]: ExpandBatchNorm finished after 0.536 seconds
2024-08-15T13:55:58Z INFO 1454 [sg0000/Tensorizer/TCTransform]: Running TCTransform
2024-08-15T13:55:59Z INFO 1454 [TCTransform]: Finished (changed=False)
2024-08-15T13:55:59Z INFO 1454 [sg0000/Tensorizer/TCTransform]: TCTransform finished after 0.500 seconds
2024-08-15T13:55:59Z INFO 1454 [sg0000/Tensorizer/EliminateDivs]: Running EliminateDivs
2024-08-15T13:55:59Z INFO 1454 [EliminateDivs]: Finished (changed=False)
2024-08-15T13:55:59Z INFO 1454 [sg0000/Tensorizer/EliminateDivs]: EliminateDivs finished after 0.432 seconds
2024-08-15T13:55:59Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: Running GenericAccessSimplifier
2024-08-15T13:56:00Z INFO 1454 [GenericAccessSimplifier]: Finished (changed=False)
2024-08-15T13:56:00Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: GenericAccessSimplifier finished after 0.501 seconds
2024-08-15T13:56:00Z INFO 1454 [sg0000/Tensorizer/TensorOpTransform]: Running TensorOpTransform
2024-08-15T13:56:02Z INFO 1454 [TensorOpTransform]: Finished (changed=True)
2024-08-15T13:56:02Z INFO 1454 [sg0000/Tensorizer/TensorOpTransform]: TensorOpTransform finished after 1.856 seconds
2024-08-15T13:56:02Z INFO 1454 [sg0000/Tensorizer/LateLowerTensorOp]: Running LateLowerTensorOp
2024-08-15T13:56:02Z INFO 1454 [LateLowerTensorOp]: Finished (changed=True)
2024-08-15T13:56:02Z INFO 1454 [sg0000/Tensorizer/LateLowerTensorOp]: LateLowerTensorOp finished after 0.731 seconds
2024-08-15T13:56:02Z INFO 1454 [sg0000/Tensorizer/MemcpyElimination]: Running MemcpyElimination
2024-08-15T13:56:13Z INFO 1454 [MemcpyElimination]: Finished (changed=True)
2024-08-15T13:56:13Z INFO 1454 [sg0000/Tensorizer/MemcpyElimination]: MemcpyElimination finished after 10.702 seconds
2024-08-15T13:56:13Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: Running LoopFusion
2024-08-15T13:56:27Z INFO 1454 [LoopFusion]: Finished (changed=True)
2024-08-15T13:56:28Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: LoopFusion finished after 14.471 seconds
2024-08-15T13:56:28Z INFO 1454 [sg0000/Tensorizer/Rematerialization]: Running Rematerialization
2024-08-15T13:56:28Z INFO 1454 [Rematerialization]: Finished (changed=True)
2024-08-15T13:56:28Z INFO 1454 [sg0000/Tensorizer/Rematerialization]: Rematerialization finished after 0.975 seconds
2024-08-15T13:56:28Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:56:30Z INFO 1454 [Simplifier]: Finished (changed=True)
2024-08-15T13:56:30Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 1.616 seconds
2024-08-15T13:56:30Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Running Delinearization
2024-08-15T13:56:32Z INFO 1454 [Delinearization]: Finished (changed=True)
2024-08-15T13:56:32Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Delinearization finished after 1.613 seconds
2024-08-15T13:56:32Z INFO 1454 [sg0000/Tensorizer/AliasDependencyElimination]: Running AliasDependencyElimination
2024-08-15T13:56:32Z INFO 1454 [AliasDependencyElimination]: Finished (changed=False)
2024-08-15T13:56:32Z INFO 1454 [sg0000/Tensorizer/AliasDependencyElimination]: AliasDependencyElimination finished after 0.332 seconds
2024-08-15T13:56:32Z INFO 1454 [sg0000/Tensorizer/DeadStoreElimination]: Running DeadStoreElimination
2024-08-15T13:56:38Z INFO 1454 [DeadStoreElimination]: Finished (changed=True)
2024-08-15T13:56:38Z INFO 1454 [sg0000/Tensorizer/DeadStoreElimination]: DeadStoreElimination finished after 5.731 seconds
2024-08-15T13:56:38Z INFO 1454 [sg0000/Tensorizer/AliasDependencyInduction]: Running AliasDependencyInduction
2024-08-15T13:56:38Z INFO 1454 [AliasDependencyInduction]: Finished (changed=True)
2024-08-15T13:56:38Z INFO 1454 [sg0000/Tensorizer/AliasDependencyInduction]: AliasDependencyInduction finished after 0.078 seconds
2024-08-15T13:56:38Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:56:39Z INFO 1454 [Simplifier]: Finished (changed=False)
2024-08-15T13:56:39Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 0.666 seconds
2024-08-15T13:56:39Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:56:39Z INFO 1454 [LICM]: Finished (changed=True)
2024-08-15T13:56:39Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.541 seconds
2024-08-15T13:56:39Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Running Delinearization
2024-08-15T13:56:40Z INFO 1454 [Delinearization]: Finished (changed=False)
2024-08-15T13:56:40Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Delinearization finished after 0.594 seconds
2024-08-15T13:56:40Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: Running LoopFusion
2024-08-15T13:56:43Z INFO 1454 [LoopFusion]: Finished (changed=True)
2024-08-15T13:56:43Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: LoopFusion finished after 3.236 seconds
2024-08-15T13:56:43Z INFO 1454 [sg0000/Tensorizer/SimplifySlice]: Running SimplifySlice
2024-08-15T13:56:43Z INFO 1454 [SimplifySlice]: Finished (changed=False)
2024-08-15T13:56:43Z INFO 1454 [sg0000/Tensorizer/SimplifySlice]: SimplifySlice finished after 0.300 seconds
2024-08-15T13:56:43Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:56:44Z INFO 1454 [LICM]: Finished (changed=True)
2024-08-15T13:56:44Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.475 seconds
2024-08-15T13:56:44Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:56:45Z INFO 1454 [Simplifier]: Finished (changed=True)
2024-08-15T13:56:45Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 1.352 seconds
2024-08-15T13:56:45Z INFO 1454 [sg0000/Tensorizer/ValueNumbering]: Running ValueNumbering
2024-08-15T13:56:46Z INFO 1454 [ValueNumbering]: Finished (changed=True)
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/ValueNumbering]: ValueNumbering finished after 0.672 seconds
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:56:46Z INFO 1454 [LICM]: Finished (changed=False)
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.441 seconds
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/PadElimination]: Running PadElimination
2024-08-15T13:56:46Z INFO 1454 [PadElimination]: Finished (changed=False)
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/PadElimination]: PadElimination finished after 0.016 seconds
2024-08-15T13:56:46Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Running Delinearization
2024-08-15T13:56:47Z INFO 1454 [Delinearization]: Finished (changed=False)
2024-08-15T13:56:47Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Delinearization finished after 0.624 seconds
2024-08-15T13:56:47Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: Running LoopFusion
2024-08-15T13:56:49Z INFO 1454 [LoopFusion]: Finished (changed=False)
2024-08-15T13:56:49Z INFO 1454 [sg0000/Tensorizer/LoopFusion]: LoopFusion finished after 2.578 seconds
2024-08-15T13:56:49Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: Running GenericAccessSimplifier
2024-08-15T13:56:50Z INFO 1454 [GenericAccessSimplifier]: Finished (changed=True)
2024-08-15T13:56:50Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: GenericAccessSimplifier finished after 0.321 seconds
2024-08-15T13:56:50Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:56:51Z INFO 1454 [Simplifier]: Finished (changed=True)
2024-08-15T13:56:51Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 1.359 seconds
2024-08-15T13:56:51Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:56:52Z INFO 1454 [LICM]: Finished (changed=True)
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.462 seconds
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/ValueNumbering]: Running ValueNumbering
2024-08-15T13:56:52Z INFO 1454 [ValueNumbering]: Finished (changed=False)
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/ValueNumbering]: ValueNumbering finished after 0.580 seconds
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/TCTransform]: Running TCTransform
2024-08-15T13:56:52Z INFO 1454 [TCTransform]: Finished (changed=False)
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/TCTransform]: TCTransform finished after 0.318 seconds
2024-08-15T13:56:52Z INFO 1454 [sg0000/Tensorizer/CommuteConcat]: Running CommuteConcat
2024-08-15T13:56:53Z INFO 1454 [CommuteConcat]: Finished (changed=False)
2024-08-15T13:56:53Z INFO 1454 [sg0000/Tensorizer/CommuteConcat]: CommuteConcat finished after 0.329 seconds
2024-08-15T13:56:53Z INFO 1454 [sg0000/Tensorizer/RecognizeOpIdiom]: Running RecognizeOpIdiom
2024-08-15T13:56:54Z INFO 1454 [RecognizeOpIdiom]: Finished (changed=False)
2024-08-15T13:56:54Z INFO 1454 [sg0000/Tensorizer/RecognizeOpIdiom]: RecognizeOpIdiom finished after 0.893 seconds
2024-08-15T13:56:54Z INFO 1454 [sg0000/Tensorizer/MaskPropagation]: Running MaskPropagation
2024-08-15T13:56:55Z INFO 1454 [MaskPropagation]: Finished (changed=False)
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/MaskPropagation]: MaskPropagation finished after 1.081 seconds
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/Recompute]: Running Recompute
2024-08-15T13:56:55Z INFO 1454 [Recompute]: Finished (changed=False)
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/Recompute]: Recompute finished after 0.036 seconds
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/DeadCodeElimination]: Running DeadCodeElimination
2024-08-15T13:56:55Z INFO 1454 [DeadCodeElimination]: Finished (changed=False)
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/DeadCodeElimination]: DeadCodeElimination finished after 0.354 seconds
2024-08-15T13:56:55Z INFO 1454 [Tensorizer]: After optimization: 2913 statements
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/DoNothing]: Running DoNothing
2024-08-15T13:56:55Z INFO 1454 [DoNothing]: Finished (changed=True)
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/DoNothing]: DoNothing finished after 0.000 seconds
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/MutateDataType]: Running MutateDataType
2024-08-15T13:56:55Z INFO 1454 [MutateDataType]: Finished (changed=True)
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/MutateDataType]: MutateDataType finished after 0.244 seconds
2024-08-15T13:56:55Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: Running GenericAccessSimplifier
2024-08-15T13:56:56Z INFO 1454 [GenericAccessSimplifier]: Finished (changed=False)
2024-08-15T13:56:56Z INFO 1454 [sg0000/Tensorizer/GenericAccessSimplifier]: GenericAccessSimplifier finished after 0.328 seconds
2024-08-15T13:56:56Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Running Simplifier
2024-08-15T13:56:57Z INFO 1454 [Simplifier]: Finished (changed=True)
2024-08-15T13:56:57Z INFO 1454 [sg0000/Tensorizer/Simplifier]: Simplifier finished after 1.645 seconds
2024-08-15T13:56:57Z INFO 1454 [sg0000/Tensorizer/AliasDependencyElimination]: Running AliasDependencyElimination
2024-08-15T13:56:58Z INFO 1454 [AliasDependencyElimination]: Finished (changed=True)
2024-08-15T13:56:58Z INFO 1454 [sg0000/Tensorizer/AliasDependencyElimination]: AliasDependencyElimination finished after 0.337 seconds
2024-08-15T13:56:58Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: Running DelinearIndices
2024-08-15T13:56:59Z INFO 1454 [DelinearIndices]: Finished (changed=True)
2024-08-15T13:56:59Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: DelinearIndices finished after 1.405 seconds
2024-08-15T13:56:59Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Running Delinearization
2024-08-15T13:57:00Z INFO 1454 [Delinearization]: Finished (changed=False)
2024-08-15T13:57:00Z INFO 1454 [sg0000/Tensorizer/Delinearization]: Delinearization finished after 0.613 seconds
2024-08-15T13:57:00Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: Running DelinearIndices
2024-08-15T13:57:01Z INFO 1454 [DelinearIndices]: Finished (changed=False)
2024-08-15T13:57:01Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: DelinearIndices finished after 1.361 seconds
2024-08-15T13:57:01Z INFO 1454 [sg0000/Tensorizer/DeadCodeElimination]: Running DeadCodeElimination
2024-08-15T13:57:02Z INFO 1454 [DeadCodeElimination]: Finished (changed=False)
2024-08-15T13:57:02Z INFO 1454 [sg0000/Tensorizer/DeadCodeElimination]: DeadCodeElimination finished after 0.340 seconds
2024-08-15T13:57:02Z INFO 1454 [sg0000/Tensorizer/InferIntrinsicOnCC]: Running InferIntrinsicOnCC
2024-08-15T13:57:04Z INFO 1454 [InferIntrinsicOnCC]: Finished (changed=True)
2024-08-15T13:57:04Z INFO 1454 [sg0000/Tensorizer/InferIntrinsicOnCC]: InferIntrinsicOnCC finished after 2.726 seconds
2024-08-15T13:57:04Z INFO 1454 [sg0000/Tensorizer/ResolveAccessConflict]: Running ResolveAccessConflict
2024-08-15T13:57:06Z INFO 1454 [ResolveAccessConflict]: Finished (changed=True)
2024-08-15T13:57:06Z INFO 1454 [sg0000/Tensorizer/ResolveAccessConflict]: ResolveAccessConflict finished after 1.700 seconds
2024-08-15T13:57:06Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:57:06Z INFO 1454 [LICM]: Finished (changed=True)
2024-08-15T13:57:06Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.512 seconds
2024-08-15T13:57:06Z INFO 1454 [sg0000/Tensorizer/LocalLayoutOpt]: Running LocalLayoutOpt
2024-08-15T13:57:14Z INFO 1454 [LocalLayoutOpt]: Finished (changed=True)
2024-08-15T13:57:14Z INFO 1454 [sg0000/Tensorizer/LocalLayoutOpt]: LocalLayoutOpt finished after 7.053 seconds
2024-08-15T13:57:14Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: Running DelinearIndices
2024-08-15T13:57:15Z INFO 1454 [DelinearIndices]: Finished (changed=True)
2024-08-15T13:57:15Z INFO 1454 [sg0000/Tensorizer/DelinearIndices]: DelinearIndices finished after 1.408 seconds
2024-08-15T13:57:15Z INFO 1454 [sg0000/Tensorizer/OrigLayoutTilingPipeline]: Running OrigLayoutTilingPipeline
2024-08-15T13:57:15Z INFO 1454 [sg0000/Tensorizer/GlobalLayoutOpt]: Running GlobalLayoutOpt
2024-08-15T13:58:34Z INFO 1454 [GlobalLayoutOpt]: Finished (changed=True)
2024-08-15T13:58:34Z INFO 1454 [sg0000/Tensorizer/GlobalLayoutOpt]: GlobalLayoutOpt finished after 79.484 seconds
2024-08-15T13:58:34Z INFO 1454 [sg0000/Tensorizer/CanonicalizeDAG]: Running CanonicalizeDAG
2024-08-15T13:58:36Z INFO 1454 [CanonicalizeDAG]: Finished (changed=True)
2024-08-15T13:58:36Z INFO 1454 [sg0000/Tensorizer/CanonicalizeDAG]: CanonicalizeDAG finished after 1.119 seconds
2024-08-15T13:58:36Z INFO 1454 [sg0000/Tensorizer/FlattenAxesForTiling]: Running FlattenAxesForTiling
2024-08-15T13:58:36Z INFO 1454 [FlattenAxesForTiling]: Finished (changed=True)
2024-08-15T13:58:36Z INFO 1454 [sg0000/Tensorizer/FlattenAxesForTiling]: FlattenAxesForTiling finished after 0.898 seconds
2024-08-15T13:58:36Z INFO 1454 [sg0000/Tensorizer/SundaSizeTiling]: Running SundaSizeTiling
2024-08-15T13:59:14Z INFO 1454 [SundaSizeTiling]: Finished (changed=True)
2024-08-15T13:59:14Z INFO 1454 [sg0000/Tensorizer/SundaSizeTiling]: SundaSizeTiling finished after 37.993 seconds
2024-08-15T13:59:14Z INFO 1454 [sg0000/Tensorizer/OrigLayoutTilingPipeline]: OrigLayoutTilingPipeline finished after 119.518 seconds
2024-08-15T13:59:14Z INFO 1454 [sg0000/Tensorizer/TilingProfiler]: Running TilingProfiler
2024-08-15T13:59:18Z INFO 1454 [TilingProfiler]: Finished (changed=False)
2024-08-15T13:59:18Z INFO 1454 [sg0000/Tensorizer/TilingProfiler]: TilingProfiler finished after 3.099 seconds
2024-08-15T13:59:18Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: Running FlattenMacroLoop
2024-08-15T13:59:21Z INFO 1454 [FlattenMacroLoop]: Finished (changed=True)
2024-08-15T13:59:21Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: FlattenMacroLoop finished after 3.167 seconds
2024-08-15T13:59:21Z INFO 1454 [sg0000/Tensorizer/InferNeuronTensor]: Running InferNeuronTensor
2024-08-15T13:59:35Z INFO 1454 [InferNeuronTensor]: Finished (changed=True)
2024-08-15T13:59:35Z INFO 1454 [sg0000/Tensorizer/InferNeuronTensor]: InferNeuronTensor finished after 14.245 seconds
2024-08-15T13:59:35Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: Running NeuronSimplifier
2024-08-15T13:59:38Z INFO 1454 [NeuronSimplifier]: Finished (changed=False)
2024-08-15T13:59:38Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: NeuronSimplifier finished after 2.972 seconds
2024-08-15T13:59:38Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T13:59:39Z INFO 1454 [LICM]: Finished (changed=True)
2024-08-15T13:59:39Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 0.983 seconds
2024-08-15T13:59:39Z INFO 1454 [sg0000/Tensorizer/RewriteReplicationMatmul]: Running RewriteReplicationMatmul
2024-08-15T13:59:40Z INFO 1454 [RewriteReplicationMatmul]: Finished (changed=False)
2024-08-15T13:59:40Z INFO 1454 [sg0000/Tensorizer/RewriteReplicationMatmul]: RewriteReplicationMatmul finished after 0.716 seconds
2024-08-15T13:59:40Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: Running FlattenMacroLoop
2024-08-15T13:59:42Z INFO 1454 [FlattenMacroLoop]: Finished (changed=True)
2024-08-15T13:59:42Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: FlattenMacroLoop finished after 2.467 seconds
2024-08-15T13:59:42Z INFO 1454 [sg0000/Tensorizer/SimplifyMacroPredicates]: Running SimplifyMacroPredicates
2024-08-15T13:59:43Z INFO 1454 [SimplifyMacroPredicates]: Finished (changed=True)
2024-08-15T13:59:43Z INFO 1454 [sg0000/Tensorizer/SimplifyMacroPredicates]: SimplifyMacroPredicates finished after 1.305 seconds
2024-08-15T13:59:43Z INFO 1454 [sg0000/Tensorizer/DataLocalityOpt]: Running DataLocalityOpt
2024-08-15T13:59:51Z INFO 1454 [DataLocalityOpt]: Finished (changed=True)
2024-08-15T13:59:51Z INFO 1454 [sg0000/Tensorizer/DataLocalityOpt]: DataLocalityOpt finished after 7.071 seconds
2024-08-15T13:59:51Z INFO 1454 [sg0000/Tensorizer/DMATilingProfiler]: Running DMATilingProfiler
2024-08-15T13:59:51Z INFO 1454 [DMATilingProfiler]: Finished (changed=False)
2024-08-15T13:59:51Z INFO 1454 [sg0000/Tensorizer/DMATilingProfiler]: DMATilingProfiler finished after 0.711 seconds
2024-08-15T13:59:51Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: Running NeuronSimplifier
2024-08-15T13:59:54Z INFO 1454 [NeuronSimplifier]: Finished (changed=False)
2024-08-15T13:59:54Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: NeuronSimplifier finished after 2.870 seconds
2024-08-15T13:59:54Z INFO 1454 [sg0000/Tensorizer/LegalizeSundaMacro]: Running LegalizeSundaMacro
2024-08-15T13:59:56Z INFO 1454 [LegalizeSundaMacro]: Finished (changed=True)
2024-08-15T13:59:56Z INFO 1454 [sg0000/Tensorizer/LegalizeSundaMacro]: LegalizeSundaMacro finished after 2.227 seconds
2024-08-15T13:59:56Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: Running NeuronSimplifier
2024-08-15T13:59:59Z INFO 1454 [NeuronSimplifier]: Finished (changed=False)
2024-08-15T13:59:59Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: NeuronSimplifier finished after 3.022 seconds
2024-08-15T13:59:59Z INFO 1454 [sg0000/Tensorizer/PerfectLoopNest]: Running PerfectLoopNest
2024-08-15T14:00:00Z INFO 1454 [PerfectLoopNest]: Finished (changed=False)
2024-08-15T14:00:00Z INFO 1454 [sg0000/Tensorizer/PerfectLoopNest]: PerfectLoopNest finished after 0.738 seconds
2024-08-15T14:00:00Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: Running FlattenMacroLoop
2024-08-15T14:00:01Z INFO 1454 [FlattenMacroLoop]: Finished (changed=True)
2024-08-15T14:00:01Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: FlattenMacroLoop finished after 0.915 seconds
2024-08-15T14:00:01Z INFO 1454 [sg0000/Tensorizer/RewriteWeights]: Running RewriteWeights
2024-08-15T14:00:02Z INFO 1454 [RewriteWeights]: Finished (changed=True)
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/RewriteWeights]: RewriteWeights finished after 0.578 seconds
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/ReshapeWeights]: Running ReshapeWeights
2024-08-15T14:00:02Z INFO 1454 [ReshapeWeights]: Finished (changed=True)
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/ReshapeWeights]: ReshapeWeights finished after 0.142 seconds
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: Running FlattenMacroLoop
2024-08-15T14:00:02Z INFO 1454 [FlattenMacroLoop]: Finished (changed=False)
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/FlattenMacroLoop]: FlattenMacroLoop finished after 0.664 seconds
2024-08-15T14:00:02Z INFO 1454 [sg0000/Tensorizer/SimplifyMacroPredicates]: Running SimplifyMacroPredicates
2024-08-15T14:00:04Z INFO 1454 [SimplifyMacroPredicates]: Finished (changed=True)
2024-08-15T14:00:04Z INFO 1454 [sg0000/Tensorizer/SimplifyMacroPredicates]: SimplifyMacroPredicates finished after 1.494 seconds
2024-08-15T14:00:04Z INFO 1454 [sg0000/Tensorizer/InferInitValue]: Running InferInitValue
2024-08-15T14:00:11Z INFO 1454 [InferInitValue]: Finished (changed=True)
2024-08-15T14:00:11Z INFO 1454 [sg0000/Tensorizer/InferInitValue]: InferInitValue finished after 7.520 seconds
2024-08-15T14:00:11Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: Running NeuronSimplifier
2024-08-15T14:00:14Z INFO 1454 [NeuronSimplifier]: Finished (changed=False)
2024-08-15T14:00:14Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifier]: NeuronSimplifier finished after 2.927 seconds
2024-08-15T14:00:14Z INFO 1454 [sg0000/Tensorizer/SimplifyTensor]: Running SimplifyTensor
2024-08-15T14:00:17Z INFO 1454 [SimplifyTensor]: Finished (changed=True)
2024-08-15T14:00:17Z INFO 1454 [sg0000/Tensorizer/SimplifyTensor]: SimplifyTensor finished after 2.175 seconds
2024-08-15T14:00:17Z INFO 1454 [sg0000/Tensorizer/LICM]: Running LICM
2024-08-15T14:00:18Z INFO 1454 [LICM]: Finished (changed=False)
2024-08-15T14:00:18Z INFO 1454 [sg0000/Tensorizer/LICM]: LICM finished after 1.049 seconds
2024-08-15T14:00:18Z INFO 1454 [sg0000/Tensorizer/SundaISel]: Running SundaISel
2024-08-15T14:00:26Z INFO 1454 [SundaISel]: Finished (changed=True)
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/SundaISel]: SundaISel finished after 7.884 seconds
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/PreprocessNkiKernels]: Running PreprocessNkiKernels
2024-08-15T14:00:26Z INFO 1454 [PreprocessNkiKernels]: Finished (changed=False)
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/PreprocessNkiKernels]: PreprocessNkiKernels finished after 0.437 seconds
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/NeuronLoopInterchange]: Running NeuronLoopInterchange
2024-08-15T14:00:26Z INFO 1454 [NeuronLoopInterchange]: Finished (changed=True)
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/NeuronLoopInterchange]: NeuronLoopInterchange finished after 0.445 seconds
2024-08-15T14:00:26Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifyPredicates]: Running NeuronSimplifyPredicates
2024-08-15T14:00:27Z INFO 1454 [NeuronSimplifyPredicates]: Finished (changed=False)
2024-08-15T14:00:27Z INFO 1454 [sg0000/Tensorizer/NeuronSimplifyPredicates]: NeuronSimplifyPredicates finished after 0.519 seconds
2024-08-15T14:00:27Z INFO 1454 [sg0000/Tensorizer/NeuronLoopFusion]: Running NeuronLoopFusion
2024-08-15T14:00:52Z INFO 1454 [NeuronLoopFusion]: Finished (changed=True)
2024-08-15T14:00:52Z INFO 1454 [sg0000/Tensorizer/NeuronLoopFusion]: NeuronLoopFusion finished after 24.729 seconds
2024-08-15T14:00:52Z INFO 1454 [sg0000/Tensorizer/NeuronLoopInterchange]: Running NeuronLoopInterchange
2024-08-15T14:00:52Z INFO 1454 [NeuronLoopInterchange]: Finished (changed=True)
2024-08-15T14:00:52Z INFO 1454 [sg0000/Tensorizer/NeuronLoopInterchange]: NeuronLoopInterchange finished after 0.378 seconds
2024-08-15T14:00:52Z INFO 1454 [sg0000/Tensorizer/NeuronLICM]: Running NeuronLICM
2024-08-15T14:00:53Z INFO 1454 [NeuronLICM]: Finished (changed=True)
2024-08-15T14:00:54Z INFO 1454 [sg0000/Tensorizer/NeuronLICM]: NeuronLICM finished after 1.409 seconds
2024-08-15T14:00:54Z INFO 1454 [sg0000/Tensorizer/FactorizeBlkDims]: Running FactorizeBlkDims
2024-08-15T14:01:03Z INFO 1454 [FactorizeBlkDims]: Finished (changed=True)
2024-08-15T14:01:03Z INFO 1454 [sg0000/Tensorizer/FactorizeBlkDims]: FactorizeBlkDims finished after 9.859 seconds
2024-08-15T14:01:03Z INFO 1454 [sg0000/Tensorizer/NeuronInstComb]: Running NeuronInstComb
2024-08-15T14:01:18Z INFO 1454 [NeuronInstComb]: Finished (changed=True)
2024-08-15T14:01:18Z INFO 1454 [sg0000/Tensorizer/NeuronInstComb]: NeuronInstComb finished after 14.606 seconds
2024-08-15T14:01:18Z INFO 1454 [sg0000/Tensorizer/NeuronValueNumbering]: Running NeuronValueNumbering
2024-08-15T14:01:19Z INFO 1454 [sg0000/Tensorizer/NeuronValueNumbering]: NeuronValueNumbering finished after 1.261 seconds
2024-08-15T14:01:19Z ERROR 1454 [Tensorizer]: Transformation error on operator: aten__add_add.31237
2024-08-15T14:01:19Z ERROR 1454 [NeuronAssert]: Assertion failure in neuronxcc/starfish/penguin/ir/IRBuilder.py:388:insertAtEnd(): incompatible function arguments. The following argument types are supported:
    1. (self: neuronxcc.pelican.ir.Instruction, arg0: neuronxcc.pelican.ir.Block) -> None

Invoked with: I-78937, 0
2024-08-15T14:01:19Z USER 1454 [root/Tensorizer/Tensorizer]: Tensorizer finished after 340.341 seconds
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]: ***************************************************************
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:  An Internal Compiler Error has occurred
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]: ***************************************************************
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]: [TEN404] (aten__add_add.31237) Internal tensorizer error: NeuronValueNumbering:insertAtEnd(): incompatible function arguments. The following argument types are supported: - P
lease open a support ticket at https://github.com/aws-neuron/aws-neuron-sdk/issues/new. You may also be able to obtain more information using the 'XLA_IR_DEBUG' and 'XLA_HLO_DEBUG' environment variables.
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]: Internal details:
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]: Type: <class 'neuronxcc.logging.Assert.NeuronAssertionError'>
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/CommandDriver.py", line 343, in neuronxcc.driver.CommandDriver.CommandDriver.run_subcommand
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/commands/CompileCommand.py", line 1277, in neuronxcc.driver.commands.CompileCommand.CompileCommand.run
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/commands/CompileCommand.py", line 1228, in neuronxcc.driver.commands.CompileCommand.CompileCommand.runPipeline
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/commands/CompileCommand.py", line 1248, in neuronxcc.driver.commands.CompileCommand.CompileCommand.runPipeline
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/commands/CompileCommand.py", line 1251, in neuronxcc.driver.commands.CompileCommand.CompileCommand.runPipeline
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/Job.py", line 346, in neuronxcc.driver.Job.SingleInputJob.run
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/Job.py", line 372, in neuronxcc.driver.Job.SingleInputJob.runOnState
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/Pipeline.py", line 30, in neuronxcc.driver.Pipeline.Pipeline.runSingleInput
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/Job.py", line 346, in neuronxcc.driver.Job.SingleInputJob.run
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/Job.py", line 372, in neuronxcc.driver.Job.SingleInputJob.runOnState
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/jobs/Frontend.py", line 431, in neuronxcc.driver.jobs.Frontend.Frontend.runSingleInput
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/jobs/Frontend.py", line 215, in neuronxcc.driver.jobs.Frontend.Frontend.runXLAFrontend
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/driver/jobs/Frontend.py", line 220, in neuronxcc.driver.jobs.Frontend.Frontend.runXLAFrontend
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Penguin.py", line 354, in neuronxcc.starfish.penguin.Penguin.runPenguin
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Frontend.py", line 155, in neuronxcc.starfish.penguin.Frontend.tensorizeXla
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Frontend.py", line 157, in neuronxcc.starfish.penguin.Frontend.tensorizeXla
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Frontend.py", line 165, in neuronxcc.starfish.penguin.Frontend.tensorizeXla
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Frontend.py", line 300, in neuronxcc.starfish.penguin.Frontend.tensorizeXlaFromFile
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Compile.py", line 266, in neuronxcc.starfish.penguin.Compile.compile_module
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Compile.py", line 269, in neuronxcc.starfish.penguin.Compile.compile_module
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/Compile.py", line 320, in neuronxcc.starfish.penguin.Compile.compile_module
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 550, in neuronxcc.starfish.penguin.DotTransform.PassManager.transformModule
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 566, in neuronxcc.starfish.penguin.DotTransform.PassManager.transformFunction
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 572, in neuronxcc.starfish.penguin.DotTransform.PassManager.transformFunction
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 170, in neuronxcc.starfish.penguin.DotTransform.DotTransform.runOnFunction
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 254, in neuronxcc.starfish.penguin.DotTransform.DotTransform.run_with_exception_handling
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 282, in neuronxcc.starfish.penguin.DotTransform.DotTransform.rethrow_exception
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/logging/Assert.py", line 92, in neuronxcc.logging.Assert.neuron_assert
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]: Cause:
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 241, in neuronxcc.starfish.penguin.DotTransform.DotTransform.run_with_exception_handling
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 296, in neuronxcc.starfish.penguin.DotTransform.DotTransform.timed_run_
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 299, in neuronxcc.starfish.penguin.DotTransform.DotTransform.timed_run_
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 300, in neuronxcc.starfish.penguin.DotTransform.DotTransform.timed_run_
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 329, in neuronxcc.starfish.penguin.DotTransform.DotTransform.run_
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 331, in neuronxcc.starfish.penguin.DotTransform.DotTransform.run_
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 449, in neuronxcc.starfish.penguin.DotTransform.DotTransform.transformFunction
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 450, in neuronxcc.starfish.penguin.DotTransform.DotTransform.transformFunction
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 441, in neuronxcc.starfish.penguin.DotTransform.DotTransform.runTransforms
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 430, in neuronxcc.starfish.penguin.DotTransform.DotTransform.transformStmts
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/DotTransform.py", line 161, in neuronxcc.starfish.penguin.DotTransform.DotTransform.transform
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/targets/tonga/passes/TongaValueNumbering.py", line 283, in neuronxcc.starfish.penguin.targets.tonga.passes.TongaValueNumbering.NeuronValue
Numbering.transformBasicBlock
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/targets/tonga/passes/TongaValueNumbering.py", line 89, in neuronxcc.starfish.penguin.targets.tonga.passes.TongaValueNumbering.coalescePart
itionBroadcastInStmt
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/common.py", line 196, in neuronxcc.starfish.penguin.common.eager_any
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/targets/tonga/passes/TongaValueNumbering.py", line 212, in neuronxcc.starfish.penguin.targets.tonga.passes.TongaValueNumbering.coalescePar
titionBroadcast
2024-08-15T14:01:19Z ERROR 1454 [neuronxcc.driver.CommandDriver]:   File "neuronxcc/starfish/penguin/ir/IRBuilder.py", line 388, in neuronxcc.starfish.penguin.ir.IRBuilder.IRBuilder.insert
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]: Diagnostic information:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   NeuronX Compiler version 2.14.227.0+2d4f85be
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   Python version 3.10.12
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   HWM version 2.14.0.227+2d4f85be
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   NumPy version 1.25.2
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   Running on AMI ami-0dd418d38400b9b7a
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:   Running in region use1-az6
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]:
2024-08-15T14:01:19Z USER 1454 [neuronxcc.driver.CommandDriver]: Diagnostic logs stored in /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/log-neuron-cc.txt
2024-08-15T14:01:19Z INFO 1454 [neuronxcc.driver.CommandDriver]: Artifacts stored in: /tmp/no-user/neuroncc_compile_workdir/174522cf-1aca-4452-8226-518f45fb1498/neuronxcc-zkczg6a7
2024-08-15T14:01:19Z INFO 1332 [root]: Subcommand returned with exitcode=70

Graph graph_cc_174522cf-1aca-4452-8226-518f45fb1498.zip

aws-taylor commented 1 month ago

Thanks @ariveram2111,

I've engaged with the relevant team and we're investigating the issue.