Single URL announcements file to communicate with downstream users and installations.

monicacecilia commented 9 years ago

From conversation at https://github.com/GMOD/Apollo/issues/190 with @kltm

Seth Carbon: "I have a rather long winded suggestion that would probably go better in its own ticket, but since I'm not sure how usable you'd find it, I figure I can test it here first as it would partially address the disruption caused here by the GO (sorry). It boils down to a single URL announcements file that apollo periodically consumes to communicate with downstream users and installations.

The idea would be that when apollo starts-up, is reloaded, or after a certain period, it checks a URL (or its backup) for a file, maybe from S3 or BBOP CDN or something. If it finds it, it parses out the sets of announcements that have a simple set of fields like:

: version range list (0.9-1.0, 2.x) : date applicability range (date when the notice should appear, can be the future for getting ahead) : announcement type (warn, notice, etc.) : announcement html

if a set passes the filters, a dismissible announcement occurs. An installation (or user) could opt-out from the system entirely in configuration.

What this would get you is:

a channel for general announcements a channel to get users to update configurations (this issue) a channel to announce outages and problems (this issue) a way of collecting passive user analytics I suggest this because you have a broad range of people with different installation types, versions, and awareness of what is going on with apollo. You may already have a system like this in place, but I thought I'd fly it since I don't know better :) "

nathandunn commented 8 years ago

@kltm has this gone anywhere (i.e., is there a URL to check?)

kltm commented 8 years ago

This was just a suggestion by me for (IIRC) a hiccup that happened in Apollo (with GO services upstream) using a strategy that was similar to something used in AmiGO. It might have passed its good-by date, but a fun idea I thought.

nathandunn commented 8 years ago

@kltm Thanks. Maybe we can revisit if we ever do get a service URL.

nathandunn commented 8 years ago

From: https://github.com/geneontology/amigo/issues/259

One idea would be to simply add the GO logo (micro-sized) to the footer, hosted on a logging resource. Or, download and toss a random resource during install.

nathandunn commented 8 years ago

The server tickle to request a central resource for when it is brought up (with a daily ping) is good (and of course configurable off). If I can version and environment (dev-mode vs production), that could be useful, and not particularly hard to add.

kltm commented 8 years ago

It would be great to coordinate on a URL structure and maybe JSON payload for this. Ideally, we could use the same announcement/ping tools and tooling in Amazon. I currently have a node tool for pulling and lightly filtering the ping log in S3. It might be nice to have a standard tool for pushing messages to S3/cloudfront. img_20161019_151950 That was the original idea (up on top there). After some further though, I think that maybe something like:

http://notice.genomearchitect.org/<all/version>.json

The idea behind the separate domain is to make sure that it can be held separately for high availability. Unlike my original idea, the URL would be simpler, with a client always trying to access two things--the payload for all clients and the payload for specific versions. There should be no error if the files are not found. The payloads would look something like:

[
  {
    "type": "(notice|warning|error)",
    "message":"text",
    "date":"2016-08-15"
  }
]

S3 has logging per object, so site data can be collected there. As well, you could give the URL an argument (a la URL?UNIQUE_ID) that would allow for (very) unique identification.

nathandunn commented 8 years ago

@kltm Interesting. Playing with that now. Looks like you are I have the "genomearchitect" bucket (unless someone else does).

A guess a few of other additional thoughts: 1 - what do the logs look like? (and how can you get the correct info / date, server ID, etc. out) 2 - how often does logging happen? 3 - is there another open-source service that already does this?

kltm commented 8 years ago

Okay, so let's see. The pattern that I'm using in AWS for AmiGO (keeping in mind that AmiGO is more centralized so the messaging system lives on disk, the ping location is external) is:

 go-amigo-usage-2.3.x/ping.json
 go-amigo-usage-2.4.x/ping.json
 go-amigo-usage-master/ping.json
 go-amigo-usage-logs/(2.3.x|2.4.x|master)/A_TON_OF_LOG_FILES_*

The */ping.json files are monitored hits, dropping them results into the log bucket.

The script to view that I currently use is in geneontology/go-site and looks like:

sjcarbon@moiraine:~/local/src/git/go-site$:) node ./scripts/install-bug-log-lister.js -f ~/local/share/secrets/bbop/aws/s3/aws-go-log-viewer.json -b go-amigo-usage-logs/master

to view "master"'s logs, with an output like:

131.243.192.111 14/Feb/2016:11:32:19 +0000
128.193.83.57   10/Feb/2016:23:28:25 +0000
131.243.192.111 21/Feb/2016:12:09:27 +0000
128.193.83.57   17/Feb/2016:21:14:18 +0000
128.3.63.137    24/Feb/2016:22:21:45 +0000
128.3.63.137    24/Feb/2016:22:21:54 +0000
128.3.63.137    24/Feb/2016:22:39:04 +0000

The actual logs are more complex--this is just the output of this tool.

Again, the AmiGO use case is slightly different right now (pinging at installation). However, I'm proposing that we could go the same direction, replacing the */ping.json with VERSION.json and all.json as outlined above, to get messages that could be picked up by all users (optional, and also useful as a deadman's announcement system in case of massive failure) and get precise site installation counts.

I think I got (1).

For (2), whenever you want, but I'd suggest at installation and every time a client completely refreshes (or periodic or something similar) so that it's useful in picking up messages.

(3) I think this is more of a very simple pattern than anything else, easily replicated in any number of fairly trivial ways. For example, one could do this just as easily with your own apache (or whatever) server. The only difference between this and a web bug is that it could be used to transmit messages. Think web SMS ;) I chose AWS for this to simplify sharing logs in a cheap, sharable (more than one user), and scalable way. Not married to it though, just the pingable message payload.

kltm commented 8 years ago

BTW, I do not have a genomearchitect bucket.

nathandunn commented 8 years ago

@kltm That would probably work just fine. I get separate IP's / servers (and potentially locations) and we can think about what level of activity, but I think what you specified makes sense. We could even ask that at install when they hit "./apollo deploy"

kltm commented 8 years ago

By ask, do you mean opt in to getting messages by polling? I guess it depends on the perceived usefulness of polling for messages and pinging. Do you have a lot of users doing secret squirrel stuff? You already have GA in there, and this is an awful lot less invasive than that.

nathandunn commented 8 years ago

True. You are right, I probably don’t need to ask, so long as there is a way to turn it off. I only wonder about IT folks flipping out because their server is sending out pings to random S3 addresses on the inter webs (as opposed to violating the clients utility), but we are doing that anyway with proxies so it seems the we are in violent agreement.

Nathan

On Oct 20, 2016, at 5:33 PM, kltm notifications@github.com wrote:

By ask, do you mean opt in to getting messages by polling? I guess it depends on the perceived usefulness of polling for messages and pinging. Do you have a lot of users doing secret squirrel stuff? You already have GA in there, and this is an awful lot less invasive than that.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/194#issuecomment-255264454, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qh5ZqsgNzQbF24UU2r_8HtAu4acCks5q2AhWgaJpZM4Dr-HK.