search

Azure / Azure-Data-Factory-Integration-Runtime-in-Windows-Container

Azure Data Factory Integration Runtime in Windows Container Sample
MIT License
25 stars 36 forks source link

Performance seems inadequate #16

Closed Alexdc-12 closed 1 year ago

Alexdc-12 commented 1 year ago

Hi When this container is installed and being used in a Synapse environment, pulling data from a DB2 database, transfer times are really slow! 700k records which is around 3GB of data takes 9 1/2 hrs to transfer. If we take docker out of the equation and install the SHIR directly on the windows server the performance issues are gone.

I am told Docker is setup to use Default process isolation mode, which should allow docker to consume as much of the hosts resources?

I also note that the stats are blank/static within the Integration Runtime monitor when using Docker, again when direct on Server they work. Memory is always 0MB, CPU stuck at 50%, Network always 0 image

As far as i am aware there is no other way to install multiple SHIRs on a VM for Synapse so i really need to get this working!

byran77 commented 1 year ago

Hi @Alexdc-12 the monitor for SHIR in container is a known issue now due to docker limitation, and thus the data on the portal is not accurate. could you please provide the ActivityID of the slow copy for us to help troubleshoot? thanks!

Alexdc-12 commented 1 year ago

Hi @Alexdc-12 the monitor for SHIR in container is a known issue now due to docker limitation, and thus the data on the portal is not accurate. could you please provide the ActivityID of the slow copy for us to help troubleshoot? thanks!

Thanks for the update, where can I keep up to date on known limitations? In terms of slow copy, since I raised this issue the problem is now resolved, I suspect it could be some internal networking that was changed!

Alexdc-12 commented 1 year ago

Hi @Alexdc-12 the monitor for SHIR in container is a known issue now due to docker limitation, and thus the data on the portal is not accurate. could you please provide the ActivityID of the slow copy for us to help troubleshoot? thanks!

Here is an ActivityID of one of the slow copies however, 6854ee8d-13c4-47be-9279-18d86c00c6fa

byran77 commented 1 year ago

Hi @Alexdc-12 the monitor for SHIR in container is a known issue now due to docker limitation, and thus the data on the portal is not accurate. could you please provide the ActivityID of the slow copy for us to help troubleshoot? thanks!

Here is an ActivityID of one of the slow copies however, 6854ee8d-13c4-47be-9279-18d86c00c6fa

glad to hear it finally works well. With the ActivityID we double-checked that there are no running errors on IR and thus the problem is transient. If the abnormal performance reduction occurs frequently, please kindly raise an Azure Support Ticket for more helps.

We are working on root problem verification of the monitor issue, and will update it on Microsoft Learn page Limitations.

Thanks!