Metric Injection Data Items

[joeledwards]

Once James' Issue (#2) is complete, the required inputs should be known, and the values which need to be passed in to the inject function of asl.seedscan.database.MetricDatabase can be added to the method signature (arguments and return).

[mikehagerty]

I pulled your latest changes to catbox2 but I am unable to connect to the database server: [java] java.sql.SQLException: No suitable driver found for jdbc:postgresql://136.177.121.210:5432/dqaOct 25, 2012 11:19:36 AM asl.seedscan.database.MetricDatabase [java] SEVERE: Could not open station database.

Is this something James is working on today ? I can wait until tomorrow to try this ...

Thanks!

-Mike

On Thu, Oct 25, 2012 at 10:33 AM, Joel Edwards notifications@github.comwrote:

Once James' Issue (#2https://github.com/joeledwards/asl-java-tools/issues/2) is complete, the required inputs should be known, and the values which need to be passed in to the inject function of asl.seedscan.database.MetricDatabase can be added to the method signature (arguments and return).

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3.

[joeledwards]

This is related to our decision to de-couple the database driver from the committed source. You can download the latest JCDB driver for PostgreSQL from the web. We are using version 9.1 of Postgres.

Downloads page: http://jdbc.postgresql.org/download.html

It is up to you whether you pull this completely out of the build process, and simply link to various jars, rather than placing these in the libs directory where they will be broken into class files, and included in the final jar.

[joeledwards]

The new class MetricInjector is a Runnable which is launched as a separate thread before running through the Scanners, and is now handed to each Scanner as it is created.

Injections are performed via a call to this class, which adds the Metric to a queue for injection. This way MetricDatabase doesn't need to worry about concurrent write requests.

[mikehagerty]

Hey guys, just a quick update/check-in.

I'm working from home today as we're awaiting the super storm that's suppose to arrive some time today and the power at my office often gets knocked out when trees are felled by the wind.

I'm working on MetricData.hashChanged() right now, computing the multiDigests for a given channel (or channel pair). When this is ready I'll clean up all the metric classes that call it, then pull the latest from Joel's Github, apply my changes and push up.

I can try to do a voice chat from my home in an hour or two.

Thanks,

-Mike

On Sat, Oct 27, 2012 at 2:45 AM, Joel Edwards notifications@github.comwrote:

The new class MetricInjector is a Runnable which is launched as a separate thread before running through the Scanners, and is now handed to each Scanner as it is created.

Injections are performed via a call to this class, which adds the Metric to a queue for injection. This way MetricDatabase doesn't need to worry about concurrent write requests.

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9833195.

[joeledwards]

James and I ran an injection test today, and confirmed that SeedScan can now push metrics into the database.

James is still working on the digest comparison SQL function. This can be used by hashChanged() when it is ready.

[jamesfholland]

I finished the comparison function. Its call is nearly identical to the other function, but without a value. It returns a boolean value. If they match it returns a TRUE, if they don't match it returns FALSE

SELECT spcomparehash( '09-17-1985', --Or any other standard date format DATE 'Metric', --Metric name VARCHAR(64) 'IU', --Network ID VARCHAR(64) 64 chars is long for network names, but the network name is stored in the same column as other groups with longer names 'STAT', --station Name VARCHAR(16) 'loc', --Location VARCHAR(16) 'ch1', --channel Name VARCHAR(16) E'\\xe4d909c290d0fb1ca068ffaddf22cbd0' -- hash in hex format );

[jamesfholland]

If there is no hash it returns false as well.

[mikehagerty]

I just finished MetricData.hashChanged() It computes the digest and puts it into a byte[] to pass back to the calling metric.

I just pushed up to joel-github and I'm about to pull everything down to catbox2 to see if the database parts still work.

-M

On Mon, Oct 29, 2012 at 3:25 PM, Joel Edwards notifications@github.comwrote:

James and I ran an injection test today, and confirmed that SeedScan can now push metrics into the database.

James is still working on the digest comparison SQL function. This can be used by hashChanged() when it is ready.

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9881922.

[jamesfholland]

Finished the get hash function SELECT spGethash( '09-17-1985', --Or any other standard date format DATE 'Metric', --Metric name VARCHAR(64) 'IU', --Network ID VARCHAR(64) 'STAT', --station Name VARCHAR(16) 'loc', --Location VARCHAR(16) 'ch1' --channel Name VARCHAR(16) );

[jamesfholland]

Renamed spGethash to spGetDigest

[jamesfholland]

Renamed spGetDigest to spGetMetricValueDigest

[mikehagerty]

I cleaned up the metric classes so they now all call MetricData.hashChanged() like: // Check to see that we have data + metadata & see if the digest has changed wrt the database: ByteBuffer digest = metricData.hashChanged(channel); or, in the case of CoherencePBM: ByteBuffer digest = metricData.hashChanged(channelX, channelY);

At the end of hashChanged() I added the logic to test the current (just) calculated digest against the digest returned from the database (e.g., this is where you will put MetricQuery() or whatever): ByteBuffer newDigest = MemberDigest.multiBuffer(digests); // This is just a place holder until the old digest can be retrieved from the database: ByteBuffer oldDigest = ByteBuffer.allocate(16);

    if (newDigest.compareTo(oldDigest) == 0){
        System.out.format("=== ByteBuffers are Equal !!\n");
        return null;
    }
    else {
        return newDigest;
    }

I guess that I'll need to supply you with a String to use to lookup the old digest inside the hashChanged() method. For now I'll add a line into each metric, something like: CoherencePBM: String dbKey = MetricResult.createResultId( new Channel("00", "LHND"), new Channel("10", "LHND") ); ByteBuffer digest = MetricData.hashChanged(channelArray, dbKey); and then inside hashChanged we'll have a line like: hashChanged(): ByteBuffer oldDigest = MetricQuery(dbKey);

Okay, I see you've got something called getMetricValueDigest, so maybe I would do something like:

hashChanged(): ByteBuffer oldDigest = getMetricValueDigest(date, metricName, station, channelId);

What I have to figure out now is how to get the metricName and station from inside MetricData.hashChanged() (these are both available inside MetricResult ...)

Let me know if I'm missing something you intended.

Thanks!

-Mike

P.S. The hurricane is upon us with full force!

On Mon, Oct 29, 2012 at 6:12 PM, James Holland notifications@github.comwrote:

Renamed spGetDigest to spGetMetricValueDigest

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9887612.

[mikehagerty]

Okay, ignore all previous mails :)

I just pushed up to the Github a version that has all the logic in place to compare the current digest to the digest stored in the database.

This happens in MetricData.hashChanged()

Right now I just have a placeholder for: //ByteBuffer oldDigest = getMetricValueDigest(date, metricName, station, channelId);

I have all the arguments (date, metricName, etc.) but within MetricData I do NOT have a handle to the database so I can't invoke this method.

I'm not sure how we want to handle this (?) We used to pass the database handle in to Scanner but this has changed to an Injector class.

Joel, can you work on this ? Once this is done we'll be able to test the newDigest against the oldDigest. After that I guess we just need to verify that Metrics are being properly injected into the database and we've got our end-to-end test ...

Thanks!

-Mike

On Mon, Oct 29, 2012 at 7:50 PM, Mike Hagerty michael.t.hagerty@gmail.comwrote:

I cleaned up the metric classes so they now all call MetricData.hashChanged() like: // Check to see that we have data + metadata & see if the digest has changed wrt the database: ByteBuffer digest = metricData.hashChanged(channel); or, in the case of CoherencePBM: ByteBuffer digest = metricData.hashChanged(channelX, channelY);

At the end of hashChanged() I added the logic to test the current (just) calculated digest against the digest returned from the database (e.g., this is where you will put MetricQuery() or whatever): ByteBuffer newDigest = MemberDigest.multiBuffer(digests); // This is just a place holder until the old digest can be retrieved from the database: ByteBuffer oldDigest = ByteBuffer.allocate(16);

    if (newDigest.compareTo(oldDigest) == 0){
        System.out.format("=== ByteBuffers are Equal !!\n");
        return null;
    }
    else {
        return newDigest;
    }

I guess that I'll need to supply you with a String to use to lookup the old digest inside the hashChanged() method. For now I'll add a line into each metric, something like: CoherencePBM: String dbKey = MetricResult.createResultId( new Channel("00", "LHND"), new Channel("10", "LHND") ); ByteBuffer digest = MetricData.hashChanged(channelArray, dbKey); and then inside hashChanged we'll have a line like: hashChanged(): ByteBuffer oldDigest = MetricQuery(dbKey);

Okay, I see you've got something called getMetricValueDigest, so maybe I would do something like:

hashChanged(): ByteBuffer oldDigest = getMetricValueDigest(date, metricName, station, channelId);

What I have to figure out now is how to get the metricName and station from inside MetricData.hashChanged() (these are both available inside MetricResult ...)

Let me know if I'm missing something you intended.

Thanks!

-Mike

P.S. The hurricane is upon us with full force!

On Mon, Oct 29, 2012 at 6:12 PM, James Holland notifications@github.comwrote:

Renamed spGetDigest to spGetMetricValueDigest

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9887612.

[joeledwards]

I pushed in changes that include a separate runnable for handling database read requests. Just realized that I forgot to place this in a thread, so that needs to be done tomorrow before we run tests.

Also important is the addition of a MetricValueIdentifier class which is used extensively for database queries. It can be created for one or two channels by calling Metric's createIdentifier() method.

The hashChanged() methods have been renamed to valueDigestChanged() for consistency. A metricDigestChanged() method may be added in the future to perform a pre-check for the entire metric before doing individual value checks. This could net some serious performance improvements, but is not a major objective.

[mikehagerty]

Joel, Looks like you were busy last night :)

I took a peek through the code this morning. One thing I notice is that this will require us to broaden our definition of what is an allowable channel. In Channel.java you'll see that I had been limiting the location code to 2 chars and the channel name to 5 chars. However, MetricResult.createChannel(MetricResult.createResultId(channelA, channelB)) will attempt to create channels like: new Channel("00-10", "LHND-LHND") and pass this to Metric.createIdentifier(channel).

Also, I see a little confusion in having getResult() methods in both Metric and MetricResult. It may be clearer to call the first one Metric.getMetricResult() since it returns a MetricResult. It looks like this method is only used once, in Scanner.

It might be time to make a tree of all the different Metric* classes to see how they interact - are we making too many of these helper classes ?

Anyhow, looks like we're almost there - good work!

I'll be in my office today if you want to do a voice chat at the usual time (?)

-Mike

On Tue, Oct 30, 2012 at 3:33 AM, Joel Edwards notifications@github.comwrote:

I pushed in changes that include a separate runnable for handling database read requests. Just realized that I forgot to place this in a thread, so that needs to be done tomorrow before we run tests.

Also important is the addition of a MetricValueIdentifier class which is used extensively for database queries. It can be created for one or two channels by calling Metric's createIdentifier() method.

The hashChanged() methods have been renamed to valueDigestChanged() for consistency. A metricDigestChanged() method may be added in the future to perform a pre-check for the entire metric before doing individual value checks. This could net some serious performance improvements, but is not a major objective.

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9896989.

[joeledwards]

Hi Mike,

I added changed to Metric.getMetricResult() and changed the Channel limitations so they are now just lower limits.

I believe there is a tool that will actually build a tree of Classes. However, it might make sense to create lower-level packages within the asl.seedscan.metrics package where various metrics reside. Maybe there could be asl.seedscan.metrics.simple, asl.seedscan.metrics.powerband, etc.

Let me know what you think

Joel

On Tue, Oct 30, 2012 at 8:17 AM, MikeHagerty notifications@github.comwrote:

Joel, Looks like you were busy last night :)

I took a peek through the code this morning. One thing I notice is that this will require us to broaden our definition of what is an allowable channel. In Channel.java you'll see that I had been limiting the location code to 2 chars and the channel name to 5 chars. However, MetricResult.createChannel(MetricResult.createResultId(channelA, channelB)) will attempt to create channels like: new Channel("00-10", "LHND-LHND") and pass this to Metric.createIdentifier(channel).

Also, I see a little confusion in having getResult() methods in both Metric and MetricResult. It may be clearer to call the first one Metric.getMetricResult() since it returns a MetricResult. It looks like this method is only used once, in Scanner.

It might be time to make a tree of all the different Metric* classes to see how they interact - are we making too many of these helper classes ?

Anyhow, looks like we're almost there - good work!

I'll be in my office today if you want to do a voice chat at the usual time (?)

-Mike

On Tue, Oct 30, 2012 at 3:33 AM, Joel Edwards notifications@github.comwrote:

I pushed in changes that include a separate runnable for handling database read requests. Just realized that I forgot to place this in a thread, so that needs to be done tomorrow before we run tests.

Also important is the addition of a MetricValueIdentifier class which is used extensively for database queries. It can be created for one or two channels by calling Metric's createIdentifier() method.

The hashChanged() methods have been renamed to valueDigestChanged() for consistency. A metricDigestChanged() method may be added in the future to perform a pre-check for the entire metric before doing individual value checks. This could net some serious performance improvements, but is not a major objective.

— Reply to this email directly or view it on GitHub< https://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9896989>.

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9907050.

[mikehagerty]

Hey Joel, Don't know if you saw my chat amendment this morning, but my suggestion is to create a VirtualChannel class that allows a wider definition of location code and channel name to accomodate the database keys (e.g., "00-10"). That way we can keep Channel in line with SEED specs (2-char location code, 5-char channel code).

Should we do a call today ?

-M

On Tue, Oct 30, 2012 at 11:24 AM, Joel Edwards notifications@github.comwrote:

Hi Mike,

I added changed to Metric.getMetricResult() and changed the Channel limitations so they are now just lower limits.

I believe there is a tool that will actually build a tree of Classes. However, it might make sense to create lower-level packages within the asl.seedscan.metrics package where various metrics reside. Maybe there could be asl.seedscan.metrics.simple, asl.seedscan.metrics.powerband, etc.

Let me know what you think

Joel

On Tue, Oct 30, 2012 at 8:17 AM, MikeHagerty notifications@github.comwrote:

Joel, Looks like you were busy last night :)

I took a peek through the code this morning. One thing I notice is that this will require us to broaden our definition of what is an allowable channel. In Channel.java you'll see that I had been limiting the location code to 2 chars and the channel name to 5 chars. However, MetricResult.createChannel(MetricResult.createResultId(channelA, channelB)) will attempt to create channels like: new Channel("00-10", "LHND-LHND") and pass this to Metric.createIdentifier(channel).

Also, I see a little confusion in having getResult() methods in both Metric and MetricResult. It may be clearer to call the first one Metric.getMetricResult() since it returns a MetricResult. It looks like this method is only used once, in Scanner.

It might be time to make a tree of all the different Metric* classes to see how they interact - are we making too many of these helper classes ?

Anyhow, looks like we're almost there - good work!

I'll be in my office today if you want to do a voice chat at the usual time (?)

-Mike

On Tue, Oct 30, 2012 at 3:33 AM, Joel Edwards notifications@github.comwrote:

I pushed in changes that include a separate runnable for handling database read requests. Just realized that I forgot to place this in a thread, so that needs to be done tomorrow before we run tests.

Also important is the addition of a MetricValueIdentifier class which is used extensively for database queries. It can be created for one or two channels by calling Metric's createIdentifier() method.

The hashChanged() methods have been renamed to valueDigestChanged() for consistency. A metricDigestChanged() method may be added in the future to perform a pre-check for the entire metric before doing individual value checks. This could net some serious performance improvements, but is not a major objective.

— Reply to this email directly or view it on GitHub<

https://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9896989>.

— Reply to this email directly or view it on GitHub< https://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9907050>.

— Reply to this email directly or view it on GitHubhttps://github.com/joeledwards/asl-java-tools/issues/3#issuecomment-9909700.

[joeledwards]

Mike, I like the idea of the VirtualChannel, it should probably just extend and overwrite the appropriate methods. This will be very useful where we want to be able to accept either. Perhaps Channel should actually be a child of VirtualChannel, since it is more strict.