Default max file size for optimize write seems to be 1 GiB, not 128MB

[KoenVerbeeck]

The page says the default file size is 128MB. However, when requesting the session config spark.microsoft.delta.optimizeWrite.binSize, the result is 1073741824, which is 1 GiB. Loading sample data to a non-partitioned table with optimize write explicitly set to true gives me a Parquet file size of 1 GiB, so the 128MB seems false.

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: c2a01590-141f-0f35-3e53-bd7b97378d03
• Version Independent ID: 0b596bc5-820b-a618-90d8-0582c16dcd85
• Content: Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics
• Content Source: articles/synapse-analytics/spark/optimize-write-for-apache-spark.md
• Service: synapse-analytics
• Sub-service: spark
• GitHub Login: @DaniBunny
• Microsoft Alias: dacoelho

[KoenVerbeeck]

At closer inspection, this page seems related to Synapse Analytics, and not Fabric, which might explain the difference. I was linked to this documentation page by a Fabric doc page which is incredibly confusing.

[ManoharLakkoju-MSFT]

@KoenVerbeeck Thanks for your feedback! We will investigate and update as appropriate.

[RamanathanChinnappan-MSFT]

@KoenVerbeeck Thanks for your contribution. Please add your ideas in below link, so our production team can review it and update the same. Ideas · Community (azure.com)