fix: Update to support up to 10K partitions

[sundar-mudupalli-work]

Hi,

Per comment, I have updated the code to support 10K partitions. Supporting more than 10K partitions is a bit more complex as explained in that comment. It turned out to be a bit more complex than mentioned in the doc because the error message with choices lists all the choices as stated here, so I have used the suggested approach for large number of values.

I suggest moving this to develop after review if it passes out integration tests. I have tested the code in my environment and it successfully produces 10K files and the first many work with data-validation configs run -cdir. So no issues there.

Thanks.

Sundar Mudupalli

[sundar-mudupalli-work]

/gcbrun

[sundar-mudupalli-work]

/gcbrun

[sundar-mudupalli-work]

/gcbrun

[sundar-mudupalli-work]

/gcbrun

[sundar-mudupalli-work]

/gcbrun