Multiple Reverb Order Sync Tasks created for same orders

[rhuckey]

We are running into this issue where there are multiple sync tasks created for the same orders. We noticed that we had over 7,000 order task synced, and upon inspection there appeared to be 50 tasks of the same order in some instances. I cleared all sync tasks, and this is what I found. First run there were three tasks synced. Three orders updated to 'shipped.' http://screencast.com/t/IDSqddp571RP

After about 5 minutes there were 3 more tasks for tasks for the same orders (total of 6). http://screencast.com/t/WTrdjD4xQj

15 minutes later there were 9 tasks for the same 3 orders. http://screencast.com/t/IDSqddp571RP

Following morning, there are 123 tasks all of the same 3 orders updated to 'shipped.' http://screencast.com/t/K5x8rPklmQsM

Can anyone explain why the module is handling orders this way? It seems that this is going to bloat the database.

[kylecrum]

@rhuckey this is correct. Fetching orders happens in cron and we purposefully fetch so that they may overlap because there are many installations where people have long-running cron jobs from other plugins / tasks. If we don't scan over a large window of orders and people's crons get backed up, then they could miss syncing some orders, which is worse than syncing the same order.

[rhuckey]

@kylecrum So is this not a potential problem with bloating the magento database or server with so many duplicate logs? What is the best practice for customers using the module? Should we clear all tasks after successfully importing new orders?

[kylecrum]

@rhuckey if clearing the tasks works for you, then that sounds like a good path to go.

[dunagan5887]

The following will be implemented to address this issue:

• A nightly cronjob will be implemented to flush completed tasks from the system (deleting them from the database)
• A configuration setting will be provided allowing for enabling/disabling this flushing
• A second configuration setting will be created to define how old a task must be (in days) in order to be flushed in this manner
• This cronjob will be set to 3:19 to avoid conflicting with myriad other cronjobs which occur on the hour or at 3:00AM

[rhuckey]

Regarding bloating - our host admin just contacted us yesterday regarding extremely large log files on the server that were eating up server space. Log files were 79Gb, most of which were reverb logs.

ls -lah |grep G

total 79G *** 15G Nov 6 16:25 exception.log *** 6.0G Nov 6 16:25 reverb_curl_requests.log *** 2.5G Nov 6 16:25 reverb_queue_processor_error.log *** 2.1G Nov 6 16:25 reverb_sync_listing_images.log *** 27G Nov 6 16:25 reverb_sync_listings_fetch.log *** 26G Nov 6 16:25 reverb_sync_listings_image.log

This doesn't seem normal. Maybe there should be some more tasks included to clear some of these as well?

[kylecrum]

@rhuckey linux and other *nix systems provide tools like logrotate that are very good at managing rotating logs and are better suited than anything we could develop.

[skwp]

@rhuckey yes we believe you should be using logrotate there to manage logs on your system. If we implement our own rotation we risk conflicting with normal tools that people use (like logrotate). Also you can turn off logs entirely. You shouldnt' need them for day to day operation in System->Configuration->Developer->Log Settings - that should help. Thanks.

[skwp]

@rhuckey we will be adding a checkbox to turn off reverb logs specifically. tracking in this issue: https://github.com/reverbdotcom/reverb-magento/issues/268

[rhuckey]

@kylecrum @skwp Thank you both. We will look into the logrotate, and I have disabled the logs in the configuration. That should make a difference.