Hello, how to get our model accelerated with Neural Engine

[yuqilol]

我们致力于讲yolov5的模型转化成coreml模型，但是在使用Neural Engine时，我们有返回结果却有报错 报错如下：

doUnloadModel:options:qos:error:: model=_ANEModel: { modelURL=file:///var/containers/Bundle/Application/F0D65F11-3592-461B-96C7-2E3A18960142/test-ai.app/best.mlmodelc/ : sourceURL= (null) : key={"isegment":1,"inputs":{"input":{"shape":[20,20,1,512,1]},"input.199":{"shape":[40,40,1,256,1]}},"outputs":{"885":{"shape":[13,1200,1,1,1]},"851":{"shape":[13,4800,1,1,1]}}} : identifierSource=0 : cacheURLIdentifier=66AF9BDE2791FC32EF6CF8235F318D4557EB049094D9DF808CCDEE5C801B3F2F_18D51104AEDB7794175450A9FB3C88E150456F8B15D562E8D04FD62B596B71D2 : string_id=0x00000000 : program=_ANEProgramForEvaluation: { programHandle=1187879380277 : intermediateBufferHandle=0 : queueDepth=127 } : state=3 : programHandle=1187879380277 : intermediateBufferHandle=0 : queueDepth=127 : attr={ ANEFModelDescription = { ANEFModelInput16KAlignmentArray = ( ); ANEFModelOutput16KAlignmentArray = ( ); ANEFModelProcedures = ( { ANEFModelInputSymbolIndexArray = ( 0, 1 ); ANEFModelOutputSymbolIndexArray = ( 0, 1 ); ANEFModelProcedureID = 0; } ); kANEFModelInputSymbolsArrayKey = ( input, "input.199" ); kANEFModelOutputSymbolsArrayKey = ( "851@output", "885@output" ); kANEFModelProcedureNameToIDMapKey = { "net_1" = 0; }; }; NetworkStatusList = ( { LiveInputList = ( { BatchStride = 655360; Batches = 1; Channels = 512; Depth = 1; DepthStride = 655360; Height = 20; Interleave = 1; Name = input; PlaneCount = 512; PlaneStride = 1280; RowStride = 64; Symbol = input; Type = Float16; Width = 20; }, { BatchStride = 1310720; Batches = 1; Channels = 256; Depth = 1; DepthStride = 1310720; Height = 40; Interleave = 1; Name = "input.199"; PlaneCount = 256; PlaneStride = 5120; RowStride = 128; Symbol = "input.199"; Type = Float16; Width = 40; } ); LiveOutputList = ( { BatchStride = 307200; Batches = 1; Channels = 1; Depth = 1; DepthStride = 307200; Height = 4800; Interleave = 1; Name = "851@output"; PlaneCount = 1; PlaneStride = 307200; RowStride = 64; Symbol = "851@output"; Type = Float16; Width = 13; }, { BatchStride = 76800; Batches = 1; Channels = 1; Depth = 1; DepthStride = 76800; Height = 1200; Interleave = 1; Name = "885@output"; PlaneCount = 1; PlaneStride = 76800; RowStride = 64; Symbol = "885@output"; Type = Float16; Width = 13; } ); Name = "net_1"; } ); } : perfStatsMask=0} was not loaded by the client. doUnloadModel:options:qos:error:: model=_ANEModel: { modelURL=file:///var/containers/Bundle/Application/F0D65F11-3592-461B-96C7-2E3A18960142/test-ai.app/best.mlmodelc/ : sourceURL= (null) : key={"isegment":0,"inputs":{"image":{"shape":[640,640,1,3,1]}},"outputs":{"input":{"shape":[20,20,1,512,1]},"y.1":{"shape":[13,80,1,80,3]},"input.199":{"shape":[40,40,1,256,1]}}} : identifierSource=0 : cacheURLIdentifier=66AF9BDE2791FC32EF6CF8235F318D4557EB049094D9DF808CCDEE5C801B3F2F_24E0A3B9B9858637900EE0297DE64AF92974CCA4DDED1627B689E2364E216DC2 : string_id=0x00000000 : program=_ANEProgramForEvaluation: { programHandle=1187873766188 : intermediateBufferHandle=1187874540920 : queueDepth=127 } : state=3 : programHandle=1187873766188 : intermediateBufferHandle=1187874540920 : queueDepth=127 : attr={ ANEFModelDescription = { ANEFModelInput16KAlignmentArray = ( ); ANEFModelOutput16KAlignmentArray = ( ); ANEFModelProcedures = ( { ANEFModelInputSymbolIndexArray = ( 0 ); ANEFModelOutputSymbolIndexArray = ( 0, 1, 2 ); ANEFModelProcedureID = 0; } ); kANEFModelInputSymbolsArrayKey = ( image ); kANEFModelOutputSymbolsArrayKey = ( "input.199@output", "input@output", "y.1@output" ); kANEFModelProcedureNameToIDMapKey = { net = 0; }; }; NetworkStatusList = ( { LiveInputList = ( { BatchStride = 2457600; Batches = 1; Channels = 3; Depth = 1; DepthStride = 2457600; Height = 640; Interleave = 1; Name = image; PlaneCount = 3; PlaneStride = 819200; RowStride = 1280; Symbol = image; Type = Float16; Width = 640; } ); LiveOutputList = ( { BatchStride = 1310720; Batches = 1; Channels = 256; Depth = 1; DepthStride = 1310720; Height = 40; Interleave = 1; Name = "input.199@output"; PlaneCount = 256; PlaneStride = 5120; RowStride = 128; Symbol = "input.199@output"; Type = Float16; Width = 40; }, { BatchStride = 655360; Batches = 1; Channels = 512; Depth = 1; DepthStride = 655360; Height = 20; Interleave = 1; Name = "input@output"; PlaneCount = 512; PlaneStride = 1280; RowStride = 64; Symbol = "input@output"; Type = Float16; Width = 20; }, { BatchStride = 409600; Batches = 3; Channels = 80; Depth = 1; DepthStride = 1228800; Height = 80; Interleave = 1; Name = "y.1@output"; PlaneCount = 240; PlaneStride = 5120; RowStride = 64; Symbol = "y.1@output"; Type = Float16; Width = 13; } ); Name = net; } ); } : perfStatsMask=0} was not loaded by the client.

[TobyRoseman]

I would start by reviewing this document: https://machinelearning.apple.com/research/neural-engine-transformers

[TobyRoseman]

I'd also recommend watching this WWDC talk: https://developer.apple.com/videos/play/wwdc2021/10038/

[ephemer]

I recently did a lot of work getting some models running on the ANE. What really helped my workflow was to convert a tiny version of the model with a similar architecture, open the resulting model in Xcode, run a performance test with CPU + ANE and then check which parts of the model run on the ANE or not. I quickly got a feel for what runs on the ANE and what doesn't (e.g. dynamism, control flow, huge tensors)