search

Azure / azure-media-migration

Azure Media Migration Tool
https://github.com/Azure/azure-media-migration
MIT License
11 stars 15 forks source link

Minimum system requirement for the tool? #246

Closed ManishKanwaria closed 3 months ago

ManishKanwaria commented 4 months ago

While testing out the migration activity, we observed that the tool works fine when the asset count is low, but when it attempts to migrate more than 20+ assets, after 20–25 minutes, the tool fails with memory exceptions, and the CPU spikes to almost 90% at this time, flagged against cmd. 

The tool is hosted on a VM with 8GB RAM configuration. 

pohhsu commented 4 months ago

Hi @ManishKanwaria,

Sorry to hear about your problem. we currently don't have a minimum system requirement published for this tool as it is dependent on your content, and also on the underlying tools it uses (FFmpeg and Shaka packager). I'm wondering if you can provide the command line you were using and the logs from the tool so we can understand your situation better.

One thing that comes to mind regarding memory usage is that the tool has a parameter called --batch-size which enables parallel processing, if you used this parameter, then the memory usage will increase. I'm not sure if you are using higher --batch-size or not.

cc @duggaraju, in case he can provide additional tips.

duggaraju commented 4 months ago

@ManishKanwaria Looking at your logs would be helpful (particularly exception stack traces). What is the typical duration of the video you are trying to migrate? Also are these live or VOD? How are you running the tool? In Azure or locally on your machine? Tracking the CPU or memory usage over time might be helpful too.

ManishKanwaria commented 4 months ago

Hi @pohhsu @duggaraju , Thank you for taking the time to look at our issue. Most of our VOD assets are, on average, 6 GB. We were persistently getting the timeout issue captured in the tool logs; hence, we thought of upgrading to a 16GB Azure VM. After that, we did not face any issues, and we were successfully able to migrate all our test assets. 

Thank you both :) 

duggaraju commented 3 months ago

The issue turned out to be a disk read limit rather than CPU/memory issue. Where the OS disk is also used as temp space and becomes the bottle neck.