hartfordfive / cloudflarebeat

ELK beat to fetch Cloudflare logs via the Enterprise Log Share API
Other
11 stars 8 forks source link

having error : ERR Could not open file for reading: gzip: invalid header #21

Open bshah0408 opened 4 years ago

bshah0408 commented 4 years ago

Hello I have been trying to find the resolution to this :

I am using cloudflarebeat-0.2.0-linux-x86_64.tar.gz

.../hartfordfive/cloudflarebeat # ./cloudflarebeat -c cloudflarebeat.yml -e -d "*" 2020/01/14 18:16:29.788932 beat.go:267: INFO Home path: [/opt/github.com/hartfordfive/cloudflarebeat] Config path: [/opt/github.com/hartfordfive/cloudflarebeat] Data path: [/opt/github.com/hartfordfive/cloudflarebeat/data] Logs path: [/opt/github.com/hartfordfive/cloudflarebeat/logs] 2020/01/14 18:16:29.788964 beat.go:177: INFO Setup Beat: cloudflarebeat; Version: 6.0.0-alpha1 2020/01/14 18:16:29.788977 processor.go:43: DBG Processors: 2020/01/14 18:16:29.788986 beat.go:183: DBG Initializing output plugins 2020/01/14 18:16:29.789153 output.go:167: INFO Loading template enabled. Reading template file: /opt/github.com/hartfordfive/cloudflarebeat/cloudflarebeat.template.json 2020/01/14 18:16:29.789594 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /opt/github.com/hartfordfive/cloudflarebeat/cloudflarebeat.template-es2x.json 2020/01/14 18:16:29.790035 client.go:120: INFO Elasticsearch url: http://0.0.0.0:9200 2020/01/14 18:16:29.790101 logp.go:219: INFO Metrics logging every 30s 2020/01/14 18:16:29.790079 outputs.go:106: INFO Activated elasticsearch as output plugin. 2020/01/14 18:16:29.790143 publish.go:234: DBG Create output worker 2020/01/14 18:16:29.790189 publish.go:276: DBG No output is defined to store the topology. The server fields might not be filled. 2020/01/14 18:16:29.790226 publish.go:291: INFO Publisher name: domain-test.com 2020/01/14 18:16:29.790413 async.go:63: INFO Flush Interval set to: 1s 2020/01/14 18:16:29.790423 async.go:64: INFO Max Bulk Size set to: 50 2020/01/14 18:16:29.790428 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50) 2020/01/14 18:16:29.790505 statefile.go:91: INFO Initializing state file 'logs-exceleron.com.state' with storage type 'disk' 2020/01/14 18:16:29.791132 statefile.go:123: INFO Saving newly initialized state file. 2020/01/14 18:16:29.791156 statefile.go:125: INFO [ERROR] Could not save new state file: open logs-exceleron.com.state: no such file or directory 2020/01/14 18:16:29.791329 beat.go:207: INFO cloudflarebeat start running. 2020/01/14 18:16:29.791341 cloudflarebeat.go:76: INFO cloudflarebeat is running! Hit CTRL-C to stop it. 2020/01/14 18:16:29.791349 cloudflarebeat.go:109: INFO Starting ticker with period of 5 minute(s) 2020/01/14 18:16:59.791236 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:17:29.791157 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:17:59.791164 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:18:29.791162 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:18:59.791156 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:19:29.791109 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:19:59.791169 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:20:29.790357 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:20:59.790284 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:21:29.791128 logp.go:232: INFO No non-zero metrics in the last 30s 2020/01/14 18:21:29.792091 cloudflarebeat.go:166: INFO Log files for time period 1579023989 to 1579024289 have been queued for download/processing. 2020/01/14 18:21:29.792107 cloudflarebeat.go:146: INFO Creating worker to publish events 2020/01/14 18:21:29.792190 logconsumer.go:61: INFO Downloading log segment #0 from 1579023989 to 1579024039 2020/01/14 18:21:29.792263 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.793917 logconsumer.go:61: INFO Downloading log segment #1 from 1579024040 to 1579024090 2020/01/14 18:21:29.793966 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.794133 logconsumer.go:61: INFO Downloading log segment #2 from 1579024091 to 1579024141 2020/01/14 18:21:29.794149 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.794234 logconsumer.go:61: INFO Downloading log segment #3 from 1579024142 to 1579024192 2020/01/14 18:21:29.794247 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.794329 logconsumer.go:61: INFO Downloading log segment #4 from 1579024193 to 1579024243 2020/01/14 18:21:29.794340 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.794422 logconsumer.go:61: INFO Downloading log segment #5 from 1579024244 to 1579024294 2020/01/14 18:21:29.794435 client.go:82: DBG Downloading log file... 2020/01/14 18:21:29.914584 client.go:97: DBG Downloaded 232 bytes 2020/01/14 18:21:29.914613 logconsumer.go:76: INFO Total download time for log file: 0 seconds 2020/01/14 18:21:29.914625 logconsumer.go:113: INFO Log file cloudflare_logs_1579024091_to_1579024141.txt.gz ready for processing. 2020/01/14 18:21:29.914640 logconsumer.go:123: INFO Opening gziped file cloudflare_logs_1579024091_to_1579024141.txt.gz for reading... 2020/01/14 18:21:29.914685 logconsumer.go:126: ERR Could not open file for reading: gzip: invalid header 2020/01/14 18:21:29.914756 client.go:97: DBG Downloaded 232 bytes 2020/01/14 18:21:29.914773 logconsumer.go:76: INFO Total download time for log file: 0 seconds 2020/01/14 18:21:29.914783 logconsumer.go:113: INFO Log file cloudflare_logs_1579024142_to_1579024192.txt.gz ready for processing. 2020/01/14 18:21:29.914796 logconsumer.go:123: INFO Opening gziped file cloudflare_logs_1579024142_to_1579024192.txt.gz for reading... 2020/01/14 18:21:29.914827 logconsumer.go:126: ERR Could not open file for reading: gzip: invalid header