search

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Enable daily only pcap processing after one historical reprocessing cycle #389

Open stephen-soltesz opened 1 year ago

stephen-soltesz commented 1 year ago

We enabled 100% pcap processing with:

Because of the pcap dataset size, this dramatically slows down all reprocessing of the etl-gardener.

With the daily only option, this can be reduced to only the daily data after the the historical pcap data has been processed once.

NOTE: by definition, this means historical pcap index parse.Time will not be updated.