timeplus-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
1 stars 0 forks source link

clean partition enhance #19

Closed yokofly closed 1 month ago

yokofly commented 1 month ago
  1. use distinct keyword to reduce duplicate parition dropped
  2. apply more common partition mode to date
    
    try use below query to match the partition daily
                SELECT DISTINCT partition
                FROM system.parts
                WHERE table = 'trade_data'
                  AND (partition LIKE '%20241015%' OR partition LIKE '%2024-10-15%' OR partition LIKE '%2024_10_15%' OR partition LIKE '%2024/10/15%')

12:48AM INF Retrieved partitions [(20241015,'SNAP')] for table: trade_data 12:48AM INF Successfully dropped partition (20241015,'SNAP') from table trade_data 12:48AM INF Finished cleaning partitions for table: trade_data



fix #18