search

Azure / azure-diagnostics-tools

Plugins and tools for collecting, processing, managing, and visualizing diagnostics data and configuration
98 stars 92 forks source link

AzureBlob work #99

Closed clguiman closed 7 years ago

clguiman commented 7 years ago
  1. Download the blob in 4MB chunks #70
  2. Add user agent
msftclas commented 7 years ago

@clguimanMSFT, Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA. Thanks, Microsoft Pull Request Bot

clguiman commented 7 years ago

This implementation will break JSON parsing. I'll need to also parse json fragments (#85) before I can merge this PR

xiaomi7732 commented 7 years ago

Generally, I would suggest we split these 2 changes to 2 PRs, they are too different. The agent string a lightweight change as long as we agreed on how to build the string, that can't break too much stuff;

The huge file issue, on the other than, is an interesting issue, it is important and could impact core functionalities. At the same time, there are interesting edge cases too, for example, what will happen if we page loading a json and the file is appended during the reading.

On the side, there are several other directions that might bring us benefits. For example, does the storage SDK supports streaming loading a blob - that we could totally avoid having the whole file loaded. Or, can we make use of the block info for azure blobs to load the file block by block.

clguiman commented 7 years ago

I've moved the user agent work in a separate PR: #101 I'll close this one and open a new PR once the partial json problem is fixed