search

gwu-libraries / social-feed-manager

"Old SFM" -- manage rules and streams from social data sources, starting with twitter.
MIT License
87 stars 20 forks source link

Getting corrupted zip files for twitter filter after stopping the filter #360

Open rajatvij opened 9 years ago

rajatvij commented 9 years ago

I am getting corrupt zip files for twitter filters after stopping the filter.

Steps to reproduce:

  1. Create and save a new active filter in SFM admin.
  2. De-activate the filter.
  3. Run python manage.py organizedata
  4. Go to the directory where zip file is stored and run gunzip -c

We can see an unexpected end of file.

NOTE: I used docker to create filter.

kerchner commented 9 years ago

@rajatvij, is this occurring in a non-docker environment?

edsu commented 9 years ago

I suspect that Supervisor is sending a signal to the filterstream process to shut it down, and it's not being handled long enough to close and flush the gzip file that is being written to, leaving it in a corrupted state. One thing you might want to try is adding a close method to sfm.ui.models.RotatingFile and then handling the interrupt signal (SIGINT, SIGTERM, I'm not sure which) when streaming the filter in sfm.ui.management.commands.filterstream so that you can call it before exiting.