search

oarevalo / BugLogHQ

BugLogHQ is a tool to centralize the handling of automated bug reports from multiple applications. BugLogHQ provides a unified view of error messages sent from any number of applications, allowing the developer to search, graph, forward, and explore the bug reports submitted by the applications.
http://www.bugloghq.com
154 stars 67 forks source link

I want to be able to tell BugLogHQ to process bugs immediately. #43

Closed cfchris closed 11 years ago

cfchris commented 11 years ago

We have a multi-tenant application where many domains point at one codebase. We have multiple IIS servers and each has a few CF servers to spread the load. All of that is load balanced with sticky session. So... when the scheduled tasks get called, there is no guarantee what instance is being hit. And bugs are staying in the queue for a long time.

I propose (and will write) an update to "notifyService" that allows you to force queue processing immediately. I will default this to "false" for backwards compatibility.

cfchris commented 11 years ago

I decided that it would be good to add this as a property that can be set via init. Then, use it as a default in "notifyService".

cfchris commented 11 years ago

I'm having a little trouble tracking down the right way to get the queue processed. I'm going to do something that works. Maybe you will be able to make it "better"....

oarevalo commented 11 years ago

Hi Chris,

I would advice against that. The server queue was implemented precisely to protect the server against rogue or over-active clients choking the server. So the idea is to maintain a sort of decoupling between the act of submitting bugs, and the internal processing of the server.

Now, with that it mind, there are a couple of things you can do:

Oscar

On Wed, May 22, 2013 at 3:36 PM, Chris Phillips notifications@github.comwrote:

I decided that it would be good to add this as a property that can be set via init. Then, use it as a default in "notifyService".

— Reply to this email directly or view it on GitHubhttps://github.com/oarevalo/BugLogHQ/issues/43#issuecomment-18312769 .

Oscar Arevalo http://about.me/oarevalo http://www.coldbricks.com - Content Management System http://www.oscararevalo.com - My Blog

oarevalo commented 11 years ago

The queue processing is triggered by the processQueue() method on the AsynchListener

https://github.com/oarevalo/BugLogHQ/blob/master/components/bugLogListenerAsync.cfc#L39

On Wed, May 22, 2013 at 3:51 PM, Chris Phillips notifications@github.comwrote:

I'm having a little trouble tracking down the right way to get the queue processed. I'm going to do something that works. Maybe you will be able to make it "better"....

— Reply to this email directly or view it on GitHubhttps://github.com/oarevalo/BugLogHQ/issues/43#issuecomment-18313394 .

Oscar Arevalo http://about.me/oarevalo http://www.coldbricks.com - Content Management System http://www.oscararevalo.com - My Blog

cfchris commented 11 years ago

Oscar, I understand the advice. I think I need to better explain my issue. I have been clicking the "process" link from the queue screen while developing. And I'm OK with the interval.

Here is the issue with our infrastructure live. We have several CF instances. I am using the bugLogListener ('bugLog.listeners.bugLogListenerWS'). So, bugs are being put into queues in 8 different CF instances. Each instance has the scheduled event. But, they are all pointed to the same URL. And based on load-balancing rules, I have no guarantee what instance they will hit. So CF instance "A1" may run it's task and the request gets processed by "B3". All 8 of our instances are calling the scheduled task every 2 minutes. But, those 8 requests could get routed in such a way, that only 3 of the instance queues get processed.

Is that any clearer? Because of that, I think it would be nice if I could choose to have them process immediately. Defaulting this to false and recommending people not use it is fine by me. But, it solves an issue we have. And it's in the pull request from my "master" branch that is pending.

oarevalo commented 11 years ago

ok, let me see if I understand correctly. You have N applications all pointing to a cluster of 8 different BugLog instances?

If that's the case then there are a couple of things that might work too.

Have you tried setting different values for the "general.externalURL" setting in the main config? By default it uses the current public url, but you can set it to each individual host. BugLog will use that to generate the URL that will be called for the scheduled task. The idea being that each BugLog instance processes its own queue.

Another thing you can try is to not use the asynch listener at all. This is controlled by the "service.serviceCFC" setting in the main config. If you change the value to "bugLog.components.bugLogListener", BugLog will process the messages as they arrive with no queueing. The asynch listener is just a wrapper around this listener that was added to have better performance; but if you are already load balancing the listeners, then each one should be able to process the messages in real-time with no problem.

Do you think that could work?

On Wed, May 22, 2013 at 4:25 PM, Chris Phillips notifications@github.comwrote:

Oscar, I understand the advice. I think I need to better explain my issue. I have been clicking the "process" link from the queue screen while developing. And I'm OK with the interval.

Here is the issue with our infrastructure live. We have several CF instances. I am using the bugLogListener ('bugLog.listeners.bugLogListenerWS'). So, bugs are being put into queues in 8 different CF instances. Each instance has the scheduled event. But, they are all pointed to the same URL. And based on load-balancing rules, I have no guarantee what instance they will hit. So CF instance "A1" may run it's task and the request gets processed by "B3". All 8 of our instances are calling the scheduled task every 2 minutes. But, those 8 requests could get routed in such a way, that only 3 of the instance queues get processed.

Is that any clearer? Because of that, I think it would be nice if I could choose to have them process immediately. Defaulting this to false and recommending people not use it is fine by me. But, it solves an issue we have. And it's in the pull request from my "master" branch that is pending.

— Reply to this email directly or view it on GitHubhttps://github.com/oarevalo/BugLogHQ/issues/43#issuecomment-18314664 .

Oscar Arevalo http://about.me/oarevalo http://www.coldbricks.com - Content Management System http://www.oscararevalo.com - My Blog

cfchris commented 11 years ago

Oscar, I set the "service.serviceCFC" to "bugLog.components.bugLogListener" and restarted CF locally. It does log bugs immediately now. And that is what I want.

However, it does throw (and log) an error if I try to click the "serviceMonitor.main" link. (error: "The method getMessageLog was not found in component bugLog.components.bugLogListener. ") Of course, I don't need to click that anymore. So, I can certainly just not click it.

However, if you look at the code that I have in the pull request. I honestly think it might be a better way. Maybe I can wrap it in a try/catch. That way it can process immediately if everything is fine, but, recover later if it's not.

What do you think?

oarevalo commented 11 years ago

I can see where is the confusion now. The bugLogService.cfc is intended to be a "stand-alone" client. That means that the idea is that you distribute that cfc (and that cfc only) with your application and provide the URL of where the actual BugLog application is located. So, bugLogService has no way of reaching the actual queue, since they are supposed to live in separate servers.

For your change to work, your app would need to be posting bug reports to a BugLog server located on the same server; and if you are doing that, then there are far better (more efficient) ways to get to BugLog than going via the URL (just instantiate the listener cfc and you are done).

Now, if your apps and buglog are actually located on separate servers, then what you want to do is invoke the queue processing on the Listener, not the client. Since listeners have direct access to the actual queue, they are the only ones that can do the processing. If you still want to be able to invoke this from within your client, then your client would have to pass the value of "autoProcessQueue" as an argument to the listener which will in turn use that to decide whether or not to invoke the queue processing.

On Wed, May 22, 2013 at 4:50 PM, Chris Phillips notifications@github.comwrote:

Oscar, I set the "service.serviceCFC" to "bugLog.components.bugLogListener" and restarted CF locally. It does log bugs immediately now. And that is what I want.

However, it does throw (and log) an error if I try to click the "serviceMonitor.main" link. (error: "The method getMessageLog was not found in component bugLog.components.bugLogListener. ") Of course, I don't need to click that anymore. So, I can certainly just not click it.

However, if you look at the code that I have in the pull request. I honestly think it might be a better way. Maybe I can wrap it in a try/catch. That way it can process immediately if everything is fine, but, recover later if it's not.

What do you think?

— Reply to this email directly or view it on GitHubhttps://github.com/oarevalo/BugLogHQ/issues/43#issuecomment-18315547 .

Oscar Arevalo http://about.me/oarevalo http://www.coldbricks.com - Content Management System http://www.oscararevalo.com - My Blog

cfchris commented 11 years ago

Here is the commit that adds autoProcessQueue. It is part of my pull request. Please take a look and comment.

I am going to go live with this. I will let you know how it goes.

cfchris commented 11 years ago

I see you logged an issue for getting rid of the error I mentioned.

If the preferred way to do immediate logging is to use "bugLog.components.bugLogListener", I will switch to that. Just let me know.

I will revert the changes from the above mentioned commit, if you feel they are a bad addition to BugLogHQ.

cfchris commented 11 years ago

I reverted my code that I had added. Hopefully you can accept my pull request now.