Implement entry qualification

[jawaad-ahmad]

I know this came up during our lessons learned last year; I don't see an issue for it yet.

It was discussed that we provide some sort of check that the students can run against to demonstrate that their application meets a minimum level of requirements and compatibility to enter into the competition. If they don't pass this, then they wouldn't be allowed to compete.

I recall we mentioned two purposes for this. First, it weeds out the entries that won't make it through the challenges early, saving us time as well as saving the students time. Second, if implemented properly, it would give the teams a level of confidence that they know their app is good enough to run and make it through the challenges.

I suppose care would be needed to ensure we don't give away too much in the implementation of the checks.

I don't recall if this was discussed or if it's my own thoughts, but I remember considering bundling this test application into the standalone RPis that are sent out to the schools. While the students have the standalone stations available for testing, they can use this application as a compatibility check.

Another option instead of or in addition to the above would be to have this test application available via the Internet the whole time so that they can point their BRATA to the correct host/port and run the check at their leisure 24/7.

I don't think we discussed design or implementation, but I'm thinking this could be one more tool on BRATA along with all their other tools. They go through their app and select something like Compatibility Check, Test Mode, or Demo Mode.

The selection could possibly prompt for a standard host/port, allowing them to enter anything we provide--whether it's on the standalone RPi, the Internet, or even some super secure very through instance we only make available at 8 a.m. on the day of the competition.

Once submitted, this could send some enter_test_mode message to the Master Server, indicating that a BRATA client is about to start API calls in demo mode, so the Master Server also knows to return canned data back rather than actually start challenges.

The BRATA application could, then, iterate through a canned set of messages, each with a canned set of parameters, and then verify the canned responses received from the MS match the expected results.

If all is good, the BRATA could display some green Passed indicator.

It could also give progress on the messages sent and tests passed, so that the students know where to concentrate their efforts if any tests don't pass.

I have this as low-priority right now because it's not necessary for the competition, but we might want to start thinking about it sooner in case it drives any other decisions we have to make.

[jawaad-ahmad]

Raised priority to medium since we lowered the one on brata-hsdc/brata.masterserver#9.

[ellerychan]

I propose providing a lower-level functionality that the MS could provide for BRATA communication testing: a web page where the user could select tests and see results of tests. Most tests would consist of single message transactions, and the MS could check that the message and the data were formatted correctly.

Another hurdle we could provide should probably be the competition day BRATA registration process. A BRATA would need to be able to register itself to enter the competition. Right now, the registration is a very simple transaction, but we could raise the bar on it.

[jawaad-ahmad]

Agreed, but...

a web page where the user could select tests and see results of tests.

For this, I imagine the students loading their BRATA on their phone, going to an HTTP page in their browser for the Master Server, and clicking around to find the page to select/run tests.

I don't see how this will help because I recall the MS didn't send messages to the phone; the phone sent messages to the MS and received responses back. Is this still true?

If so, would we instead need to have a tool or page in the BRATA app that displayed tests to select and run and see results? Then BRATA would run the selected tests, which would involve sending test messages to the configured MS and receiving responses to check and verify. This would mean no web page needed on the MS.

Right now, the registration is a very simple transaction, but we could raise the bar on it.

Are you suggesting we roll out this entry qualification in advance with one set of interactions that the students test against and code against, and then reconfigure the Master Server on competition day for different interactions--say the same messages with different or more extensive parameters--in order to see which teams' implementations are robust enough to handle the new inputs before being allowed to compete?

[ellerychan]

I don't see how this will help because I recall the MS didn't send messages to the phone; the phone sent messages to the MS and received responses back. Is this still true?

Still true, but the MS could at least provide instructions on running the tests and provide the right URLs to access, etc. Maybe activating something through the web page could put the MS in a testing mode. I haven't thought it through, but it seems easier to add stuff there than in the BRATA. You don't want the BRATA running tests, you want it running the actual student code.

Are you suggesting we roll out this entry qualification in advance with one set of interactions that the students test against and code against, and then reconfigure the Master Server on competition day for different interactions--say the same messages with different or more extensive parameters--in order to see which teams' implementations are robust enough to handle the new inputs before being allowed to compete?

I wasn't really thinking that. I was just thinking that the registration process itself could test some set of features, so that if a BRATA could actually register then it probably can work well enough to compete. For example, part of the registration transaction could have the BRATA send one of each kind of message, and the MS could check them for proper formatting. Again, still just a partially-formed idea.

[jawaad-ahmad]

I took your notes above into account in the functional spec.

Please review Sections 2.1, all of 2.2, 2.3.3, 2.4.3, 2.5.4, and 2.6.2 as well as Sections 2.9 and 2.10 for details. Please provide comments in Pull Request #97 for discussion.

[ellerychan]

I have thought about this some more, and believe there is a way to do it.

One piece of MS functionality does initiate messages: for example, it can send a Reset message to a Station. So the capability will be there, and we can drive it through a web page or by sending a REST command.

[jawaad-ahmad]

From what I recall, the stations were fine because they were on the same LAN as the MS so they provided their IP address, the MS recorded it, and then the MS used that IP address for any callbacks to the stations.

Wasn't there an issue with the phones because they were coming in from the outside over a data connection?

[ellerychan]

Wasn't there an issue with the phones because they were coming in from the outside over a data connection?

You could be right about that; but for testing BRATA functionality, the BRATA always initiates messages, so the MS would be receiving the messages and giving at minimum a pass/fail indication through the web interface. I envision a page where the user clicks a Do This Test button, and the MS provides instructions for what the BRATA should send. Then the BRATA sends it (or not) and the MS displays how it went.

If they are hitting @jaron42's MS server, we could collect some stats and be getting a good indication of the state of readiness of the teams, and maybe be alerted early to problems with the training or the specification.

[jawaad-ahmad]

Please review the sections mentioned above in the spec. I think what you just mentioned is along the same lines.

I think adding collecting stats would be a good issue to track separately. Thoughts?

[ellerychan]

I believe we are collecting information now that can be used to compile stats later. I'll leave it to you to decide about the issue tracking.

[ellerychan]

@jawaad-ahmad In the Functional Spec Sec. 2.1.2 you talk about registering alternate clients. I think that should require some manual intervention by the MS operator. It would be simplest to just invalidate the current registration, and let the Team register again with a new device. It should be sufficiently rare that it would not cause too much congestion at the officals' desk.

Alternatively, we could allow a BRATA to unregister itself, as long as it has the proper credentials. Do the competition rules allow this? If that is allowed without restrictions, then the Team could have a collection of devices that each only perform a subset of the functions needed to compete. If the intent is to have one device do it all, then a finite number of registration changes could still be enforced by the MS, but it would probably be better to use the manual approach.

Perhaps @DragonmasterJ can check on the intent.

[ellerychan]

I added Implement "Invalidate Team Registration" brata.masterserver #30 for this.

[DragonmasterJ]

I'll try to ping Mike on this.

[jaron42]

Per our team discussion we are leaving registration as a simple op like last year. For testing the plan is I have a public facing server and I give them all their own unique codes to register and keep track of who does and does not successfully register on the public server. This will give an initial test of the messaging but to the point of not getting a complex response we will also be providing on that server the Launch capability next so they can dry run doing the Launch challenge at their school. This will give them full test capability of an entire challenge including response message processing. Yes MS needs to always stay response only as our network infrastructure will not let traffic go out to the phones unsolicited, so this remains a strict design constraint.

[ellerychan]

MS needs to always stay response only as our network infrastructure will not let traffic go out to the phones unsolicited, so this remains a strict design constraint.

Does this just apply to the BRATA devices? Because we have a Reset Station message that can be sent to Pi Stations. Doesn't have to be used, but it's there.

[ellerychan]

@jaron42, @jawaad-ahmad, @DesignChallengeGuy (is it really you?)

I created a branch reg_code. I added a simple mod to ServerQueryTask.java that I think will allow the MS to use either the old authentication scheme based on _teamid or the new authentication scheme based on _regcode.

I'm not in a position to test this. I will continue working on Ian to try to get him to help out, but he is enjoying being in vacation mode.

I am submitting it as Pull Request #101 .

[jaron42]

I will pull it down to the test server today, in case it helps Mark out. I am going to go ahead and make a branch merge for the new brata project structure so we can get the new msg and audio api changes started from that.

Sent from Yahoo Mail on Android

On Sat, Dec 19, 2015 at 12:21 PM, Ellery Channotifications@github.com wrote:
@jaron42, @jawaad-ahmad, @DesignChallengeGuy (is it really you?)

I created a branch reg_code. I added a simple mod to ServerQueryTask.java that I think will allow the MS to use either the old authentication scheme based on team_id or the new authentication scheme based on reg_code.

I'm not in a position to test this. I will continue working on Ian to try to get him to help out, but he is enjoying being in vacation mode.

I am submitting it as Pull Request #101 .

— Reply to this email directly or view it on GitHub.

[mmieczkowski]

Thanks Jaron.

On Dec 20, 2015, at 11:52 AM, Jaron Blackburn notifications@github.com wrote:

I will pull it down to the test server today, in case it helps Mark out. I am going to go ahead and make a branch merge for the new brata project structure so we can get the new msg and audio api changes started from that.

Sent from Yahoo Mail on Android

On Sat, Dec 19, 2015 at 12:21 PM, Ellery Channotifications@github.com wrote: @jaron42, @jawaad-ahmad, @DesignChallengeGuy (is it really you?)

I created a branch reg_code. I added a simple mod to ServerQueryTask.java that I think will allow the MS to use either the old authentication scheme based on team_id or the new authentication scheme based on reg_code.

I'm not in a position to test this. I will continue working on Ian to try to get him to help out, but he is enjoying being in vacation mode.

I am submitting it as Pull Request #101 .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

[ellerychan]

Just so we are all on the same page: The change I made was to the Brata code, not the MS, and it should allow the Brata to work transparently with either registration scheme the MS decides to use.

I was confused by your statement about pulling it down to the test server.

I would be grateful if someone would test it to make sure the handful of lines I changed didn't break anything.