Save metadata to mongodb

[chervaliery]

For statistic purposes I need to save in my mongodb all the pastie's metadata even if the there is no match on this pastie.

This pull request allows to chose the metadata to save:

• timestamp
• url
• site

And to save or not when there is no match.

For backward compatibility, if not specified, the previous behavior is the default one.

[chervaliery]

This will be the last PR for today 😄

[cvandeplas]

Interesting. I'm wondering if this could not be merged with the save-all functionality, plus minor cleanup.

Per fetch_and_process_pastie() function; if the save-all parameter is set in the yaml, the save_pastie() function is called. This results in the paste to be saved in the archive folder, but also in the redis queue.

Looking further at the code it even seems that the redis storage is used twice for matched pasties if save-all is set (once caused by save-all, once by the match triggered by the action_on_match() function).

Would you be open to work a bit more on the PR and do some additional minor refactoring?

To summarize, my proposal:

• search_content(self) returns self.matches (or True if there are matches, or nothing)
• fetch_and_process_pastie(self) acts on the self.matches or True/False result from search_content(self) (so the if self.matches: then action_on_match(); else: action_on_miss()
• ~save-all logic is moved to action_on_miss()~ (we keep the save-all as it saves data in a different location)
• redis save logic is either moved out of the save_pastie() function to the action_on_match()/action_on_miss(), or mongo save logic is moved in the save_pastie() function.

What do you think?

[chervaliery]

Sure, I'll try to work on that. But it seems difficult to implement without breaking backward compatibility.

[cvandeplas]

Cool. thanks ! On the backward compatibility, I believe that it should only change the redis double-save effect, which I consider more as a bug than a feature. Please correct me if I'm wrong !

[chervaliery]

This is not really a bug as it saves in two different directories (dir and dir-all). So in the redis, it saves with both paths.

[cvandeplas]

ok, thanks for the great feedback. I've strikethroughed the save-all move in the upper message. Just poke me when you're ready with those few changes.

Thanks again ! and don't hesitate to shout if you need help.

[cvandeplas]

addressed by #90