search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.27k stars 21.45k forks source link

Copy from Azure Table to Data Lake Store V1 #15313

Closed ghost closed 6 years ago

ghost commented 6 years ago

The copy from azure table for most of my pipelines takes 2 -4 hours and max completes but there is only one pipeline from the same Azure Table for same duration dataset takes 9 hours and more.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

Alberto-Vega commented 6 years ago

Hi @RoopAnymous, I'm sorry to hear that you are having this issue. Question: Do all the Copy Activities have the same source-sink combination? For the title of your thread it seems like all are from Azure Table to Data Lake Store V1 according to the Performance Reference the throughput is 2 MBps per second. Please check this Performance Reference (If you haven't yet ) it has a table that shows the copy throughput number in MBps for all the source sink combinations.

Also have you tried these Performance tuning steps?

This question is a good candidate for the forums. These issues are geared towards improving the documentation. I recommend you to post a question on StackOverflow or MSDN.

Alberto-Vega commented 6 years ago

@RoopAnymous We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

ghost commented 6 years ago

Hi Alberto

I had followed the performance tuning steps but unfortunately the time did not improve. From azure table to Datalake, increasing DTU and parallel copy feature did not improved the performance. My Datafactory source and sink are in same region.

My rest of the pipes works fine but only one partition in this pipe is super slow, It is taking 16 hours to copy 31 MB of Data. In past, I configured 61 mb of data from the same partition but since the copy activity did not succeeded, I reduced the data to fetch half of the data as 31 MB still no improvement in time.

Can you please suggest here in case I am missing something?

Thanks and Regards Roopa S Dhananjaya

On Mon, Sep 24, 2018 at 8:44 AM Alberto Vega notifications@github.com wrote:

Closed #15313 https://github.com/MicrosoftDocs/azure-docs/issues/15313.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MicrosoftDocs/azure-docs/issues/15313#event-1862271743, or mute the thread https://github.com/notifications/unsubscribe-auth/AM9Z4wjHW5bZh8F__dShW_r1w0nTBx3Vks5ueE4IgaJpZM4WxQr7 .

-- Thanks and Regards Roopa

Alberto-Vega commented 6 years ago

Hi @RoopAnymous This will require some deeper analysis. Can you please file a support request @ https://aka.ms/azsupt? If you do not have access to a support plan, please reach out @ AZCommunity@microsoft.com with a link to this Doc/Issue as well as your subscription ID and we can help get the support ticket opened for this issue.