Why the fragment size is not expected with the configured size

[BijieXu]

Hi MS team,

We are using this framework developing our NIC driver. We specified the rx fragment ring size to 4096. But during run, we found the actual size of the rx fragment ring was only 128. This size was not enough for a 10G card, and as a result the performance was very low.

Following is how we configured the RxCapabilities.

rxCapabilities.FragmentBufferAlignment = 64;
rxCapabilities.FragmentRingNumberOfElementsHint = adapter->ReceiveBuffers; //4096
rxCapabilities.MaximumNumberOfQueues = adapter->RxQueueNum;
NetAdapterSetDataPathCapabilities(adapter->NetAdapter, &txCapabilities, &rxCapabilities);

I noticed the parameter "FragmentRingNumberOfElementsHint" was used in following code fragment:

perfCharacteristics.FragmentRingNumberOfElementsHint = datapathCapabilities.PreferredRxFragmentRingSize;
perfCharacteristics.MaximumFragmentBufferSize = datapathCapabilities.MaximumRxFragmentSize;
perfCharacteristics.NominalLinkSpeed = datapathCapabilities.NominalMaxRxLinkSpeed;
perfCharacteristics.MaxPacketSizeWithRsc = datapathCapabilities.MaximumRxFragmentSize + m_backfillSize;

NxPerfTunerCalculateRxParameters(&perfCharacteristics, &perfParameters);
m_rxNumPackets = perfParameters.PacketRingElementCount;
m_rxNumFragments = perfParameters.FragmentRingElementCount;
m_rxNumDataBuffers = perfParameters.NumberOfBuffers;

It seems the resulted "FragmentRingElementCount" comes from the function "NxPerfTunerCalculateRxParameters", which is not visible to us. How does it calculate the final result?

Thanks, Bijie Xu

[tylerretzlaff]

FragmentRingNumberOfElementsHint is just a hint or expression of what the driver would like to use, the framework is not required to obey it. this allows the framework the ability to arbitrate shared use of the operating system resources when they are highly contended.

internally the perf tuner logic is a work in progress and quite rudimentary. if it isn't accommodating your driver appropriately, we are interested in the circumstances where it isn't providing satisfactory resource allocation.

could you give more concrete detail related to your performance observations. it isn't clear if there is an expectation of 10Gbps performance with or without hardware offloads.

• does your driver configure offloads and the ring size is still inadequate?
• which offloads if any are configured?
• given different sets of offloads your driver is configured what approximate performance are you observing for both throughput and latency?
• is it your observation that the system is processor bound / processor utilization is high?

[BijieXu]

Hi Tyler, Thanks for the quick response. We have already given up developing our driver on CxAdapter and switched to NDIS, and we are close to pass the certification.

BR.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Tyler @.> Sent: Thursday, April 27, 2023 12:22 AM To: @.> Cc: Arthman @.>; @.> Subject: Re: [microsoft/Network-Adapter-Class-Extension] Why the fragment size is not expected with the configured size (Issue #13)

FragmentRingNumberOfElementsHint is just a hint or expression of what the driver would like to use, the framework is not required to obey it. this allows the framework the ability to arbitrate shared use of the operating system resources when they are highly contended.

internally the perf tuner logic is a work in progress and quite rudimentary. if it isn't accommodating your driver appropriately, we are interested in the circumstances where it isn't providing satisfactory resource allocation.

could you give more concrete detail related to your performance observations. it isn't clear if there is an expectation of 10Gbps performance with or without hardware offloads.

• does your driver configure offloads and the ring size is still inadequate?
• which offloads if any are configured?
• given different sets of offloads your driver is configured what approximate performance are you observing for both throughput and latency?
• is it your observation that the system is processor bound / processor utilization is high?

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/Network-Adapter-Class-Extension/issues/13#issuecomment-1523701749, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AU46NKMKC5TTJASMPZMFT2LXDFDS5ANCNFSM6AAAAAASK33N4Q. You are receiving this because you authored the thread.Message ID: @.***>

[tylerretzlaff]

Hi Tyler, Thanks for the quick response.

we're somewhat embarrassed by our response time, but we are looking forward to improving it.

We have already given up developing our driver on CxAdapter and switched to NDIS, and we are close to pass the certification.

understood, depending on the feature set required NDIS6 miniport might be an appropriate choice. for example if your device is intended to be used by server rather than workstation class devices and you want to be able to provide SR-IOV or RDMA.

if you have the time to share any points that guided you toward NDIS6 instead of NetAdapterCx we'd love to hear them, our team actually owns both frameworks so we really appreciate learning where our gaps are first hand.

[BijieXu]

Hi Tyler,

Not mean to offence you, just kidding. 😉 Yes as you mentioned, we need to support SRIOV and NetAdapterCx does not support it, which is one of the reasons we gave up NetAdapterCx. Also some of the features and enhancements are included in the preview mode and we can not use the preview mode because we need to support old windows releases.

Also we found the driver based on NetAdapterCx can not pass the HLK test case “2c_OidsNdisRequest”, and we found there is no way to get rid of that unless we changing NetAdapterCx itself.

Based on these problems and limitations, we finally decided to switch to NDIS after some struggling.

Anyway I have to say NetAdapterCx is an excellent framework, which greatly reduces the coding effort for developers. And it can handle NetBuffers with bunch of small MDL segments properly, while in which case, with NDIS we have to take care of this scenario in the driver, and we need to reorganize the NetBuffer so that our NIC firmware can handle it properly.

Looking forward to jump to NetAdapterCx at proper time in the future. BR, Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Tyler @.> Sent: Thursday, April 27, 2023 11:13 AM To: @.> Cc: Arthman @.>; @.> Subject: Re: [microsoft/Network-Adapter-Class-Extension] Why the fragment size is not expected with the configured size (Issue #13)

Hi Tyler, Thanks for the quick response.

we're somewhat embarrassed by our response time, but we are looking forward to improving it.

We have already given up developing our driver on CxAdapter and switched to NDIS, and we are close to pass the certification.

understood, depending on the feature set required NDIS6 miniport might be an appropriate choice. for example if your device is intended to be used by server rather than workstation class devices and you want to be able to provide SR-IOV or RDMA.

if you have the time to share any points that guided you toward NDIS6 instead of NetAdapterCx we'd love to hear them, our team actually owns both frameworks so we really appreciate learning where our gaps are first hand.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/Network-Adapter-Class-Extension/issues/13#issuecomment-1524574485, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AU46NKMLZMP6FA6X5Z2GVQDXDHP7BANCNFSM6AAAAAASK33N4Q. You are receiving this because you authored the thread.Message ID: @.***>

[tylerretzlaff]

thank you very much for taking the time to give the feedback. with this we'll better be able to prioritize what to focus on going forward.

thanks again.