Purge 35000 records

[ruimateus]

How can I purge 35000 records from CouchDB / Acralyzer?

Thanks

[KevinGaudin]

Wow, that is a nice number of reports. You can bulk purge from the Acralyzer Admin panel, but with such a quantity of reports this might take some time and eventually fail.

[ruimateus]

Purging does nothing, and it shows no error. Is there any solution like deleting manually on couchdb?

[KevinGaudin]

No, there is no internal CouchDB solution.

I think that it doesn't really do 'nothing', the purge is actually done in two steps:

• get the list of documents to purge from couchdb to retrieve their IDs and revision IDs
• send to CouchDB a bulk update request with IDs, revision IDs and deletion command

I think you lose patience on the first step because it really can be long when dealing with such a number of reports.

Maybe you can try increasing the number of days to keep to purge reports by chunks instead of all at once.

I was aware that this could be an issue and thinking about implementing the "bulk purge in smaller chunks" thing in the browser. I'll definitely have to do that.

If there is no way for you to do it manually, you can either:

• implement it yourself before I do and submit a Pull-Request :-D
• discard your acra-storage instance and create a new one...

Kevin

[ruimateus]

Thanks Kevin, for quick response :) How can I discard acra-storage? If I push it with erica, will it overwrite?

[ruimateus]

BTW, when I try to purge, in couchdb logs appear the following:

acra-storage/_view/reports-revision-per-day?descending=true&startkey=%5B2013,2,19%5D 404

Is this a 404? Perhaps the encoding is not right..

[KevinGaudin]

This means that you should update your acra-storage instance by getting the latest version from github and pushing it again to your couchdb. Reports are unchanged when updating a couchapp.

[ruimateus]

I cloned acra-storage, and the problem remains. I checked https://github.com/ACRA/acra-storage/tree/master/views and the view "reports-revision-per-day" doesn't exist.

I checked the last revision of acralyzer at line 200 of https://github.com/ACRA/acralyzer/blob/master/_attachments/script/service.reportsstore.js. It tries to get "reports-revision-per-day".

What am I doing wrong?

Discarding acra-storage, and creating a new one, doesn't seem like a solution to me when I want to purge.

I have about 250k-300k daily users, and the reports are so helpful..

[KevinGaudin]

Hmmm I think I refactored the name of the view while working on the aggregates branch and forgot to rename the call in the master branch.

Maybe you could instead use the acralyzer aggregates branch and tell me what you think about it ?

It should be merged in the master branch quite soon anyway.

Kevin

[KevinGaudin]

I just pushed a fix for the acralyzer master branch.

[KevinGaudin]

The same error was on the aggregates branch too. Now fixed.

[ruimateus]

Thanks. Purge by versioncode has the same problem: "reports-revision-per-app-version-code" in scripts, but view is "reports-per-app-version-code".

After your correction, can I just push the cloned couchapp? I've aprox 40k reports now, I suppose the first run will take a while, yes?

[KevinGaudin]

I'm so sorry, I actually forgot to push the changes to acra-storage I had made on my second computer. This should now be fixed, and I reverted the changes on Acralyzer.

You can get the latest revisions of both acra-storage and acralyzer and push them again to your couch instance.

When an update is done on acra-storage and you have a large number of reports, the first access to views can take a few minutes as the indexes are rebuilt.

[ruimateus]

Thank you very much! :) ACRA is so useful. Cheers!

[KevinGaudin]

Have you been able to purge your reports ? How long did it take ?

[ruimateus]

Yes, I have. About 30minutes to purge 49k reports in a Xeon E5620 by app versioncode.