Chess Annotations

[mattflaschen]

@davidkrauser and I have been discussing chess annotations. One thing that makes this unique is that PGN (Portable Game Notation), which is near-ubiquitous, can already encode almost all the relevant information (although the bare bones format is shown below). So far apppassant uses:

            "value": {
                "is_active": false,
                "pgn": "1.e4 e5 2.Nf3 d6 3.d4 Bg4 4.dxe5 Bxf3 5.Qxf3 dxe5 6.Bc4 Nf6 7.Qb3 Qe7 8.Nc3 c6 9.Bg5 b5 10.Nxb5 cxb5 11.Bxb5+ Nbd7 12.O-O-O Rd8 13.Rxd7 Rxd7 14.Rd1 Qe6 15.Bxd7+ Nxd7 16.Qb8+ Nxb8 17.Rd8# 1-0"
            }

is_active simply means whether the game is still being played, as opposed to past/historic.

Annotations for challenges (this could be shared between two player games), and potentially other issues, need to be resolved.

[ghost]

I think that we need to be able to specify the users that are participating and the originating post.

If you wanted to fork off for another game or change the players, it's as simple as making a new post with all of the previous moves, the two new users, and point the originating post field to that new post.

I'm assuming all replies to that post will constitute the moves of the game.

@mattflaschen What do you mean by challenges?

[mattflaschen]

Changing the players could be tricky, but I'm starting to agree using the initial thread post is too rigid.

A challenge is just player A challenging player B, "Would you like to play?", then B seeing this challenge (it would be an annotated mention). I'd like it if A could either pre-select who plays each color (B would see this of course, before acceptance/rejection), or select random/assigned, in which case a bot would choose.

[ghost]

What we could do for challenges, is just let the player A create the new game, specify player B, and also their color. Then, a post mentioning player B is sent out with player A's first move. Player B can decide to play, and start playing, or just ignore the post.

@mattflaschen I'm not sure is_active is a good idea, because you can't change it for previous posts. Thus, the first move will always be true for "is_active" even if the game is long over. I don't think this distinction really needs to be made. What are you using it for?

[mattflaschen]

"@mattflaschen I'm not sure is_active is a good idea, because you can't change it for previous posts. Thus, the first move will always be true for "is_active" even if the game is long over."

I don't see that as a problem. A regular post is a snapshot in time too (A said/posted X at time T1, even though he might think X is dead wrong at T2). It was active when the move/ongoing game was posted.

I'm not using it yet (except when posting), but I see at least these scenarios:

1. Share interesting games (e.g. the Opera Game), even if the game is over and wasn't played out using app.net. I've already been doing this using App Passant.
2. Actively play correspondence chess on the service.
3. Live broadcast, in which a real-life over-the-board (or off-ADN online) tournament could broadcast tournament games.

2 & 3 are active.

I'm open to a better taxonomy.

Also, I just realized having a field to pre-select the displayed move number would be useful for discussing games.

[mattflaschen]

"Then, a post mentioning player B is sent out with player A's first move"

The challenge can't include the first move if the challenger (A) wants black (this is relatively common, when someone wants practice).

[mattflaschen]

We should figure out when to use machine-only posts; they actually use chess as an example.

The simplest approach is to leave it up to the user. if they choose not to provide text (say, just a regular correspondence chess move, nothing special), mark it machine-only (machine only posts must contain no text in the regular text field).

But for some cases (e.g. challenges) we may not even want to allow text, and thus always have them be machine-only.

[ghost]

I think they should always be machine-only unless the user specifies a post. It would be nice if anyone could spectate on a game, though.

[mattflaschen]

Should we allow them to specify a post/text on all annotated posts?

[ghost]

Sure, why not? That way they can converse as they play.

Would you explain what you mean by "a field to pre-select the displayed move number" a bit more?

[mattflaschen]

When I asked (partly rhetorically) about "all annotated posts", I was thinking about challenges and such where it's more formulaic, and no moves may be involved.

"Would you explain what you mean by "a field to pre-select the displayed move number" a bit more?"

Say you observed a game (on app.net, in real life, or otherwise). You're posting the PGN of the whole game (which can itself contain embedded annotations on as many moves as you want).

However, your main point is regarding a particular move, say white's 3. So your regular app.net post text discusses that move, and it would be nice if an annotation could hint to open the chess board with the position after that move shown (clients would still allow stepping back and forth).

Just something like (using strings for numbers like app.net)

selected_ply: "5"

A ply is basically half a move (a full move being both white and black going). So white's move three would be ply 5 (1-indexed), and black's move three would be 6.

[mattflaschen]

Okay, for challenges with pre-selected colors, how about:

"value": {
    "challenge": {
        "white": "somename",
        "black": "othername"
    }
}

And it should mention the user who is not posting the challenge.

Good clients should validate that the poster is one of those users, before showing the challenge to the other user.

[mattflaschen]

This is my proposal for the rest of the game. To accept, the other player sends:

"value": {
    "challenge": {
        "white": "somename",
        "black": "othername",
        "accepted": true
    }
}

in reply to the challenge.

Before showing the acceptance to the challenger, the client should verify that:

1. The players and colors match completely.
2. The right player accepted.
3. There is no prior valid acceptance to this challenge.

A move in an ongoing game would have is_active true, and a correspondence object. Initially, that would just have white and black usernames. Later on we could add else anything specific to an active online correspondence chess game.

"value": {
    "is_active": true,
    "correspondence": {
        "white": "somename",
        "black": "othername",
        "acceptance_post_id": "123"
    },
    "pgn": "1.e4 e5 2.Nf3 d6 3.d4 Bg4 4.dxe5 Bxf3 5.Qxf3 dxe5 6.Bc4 Nf6"
}

A post should be a reply to the acceptance (first ply), or to the prior ply (any other ply). In any ply, acceptance_post_id is always the ID of the acceptance.

The PGN could optionally contain White/Black tags with the players full names, along with other relevant ones.

We should try to make it possible to validate, so serious games are feasible.

So far, I think these conditions should be true in order for a post B to be a real ply (move) in the game:

1. The poster's username matches one of the players in correspondence.
2. In the last ply of the posted PGN, they played the right color.
3. Either of:
   • This is the first ply in the PGN, and they replied to acceptance_post_id, which is a valid acceptance with the same players and colors. acceptance_post_id is a reply to a valid matching challenge.
   • (Not first) They replied to a post, with id A, that the other player made. Post A has the same acceptance_post_id.
4. There was no previous ply in reply to A that met the criteria. You can't post 3. f4, then change your mind and post 3. f3. 3. f3 (the double move) will simply be ignored.
5. The posted PGN has exactly one more ply than ply A's , or if replying to an acceptance, the PGN has exactly one ply (no tacking more than one ply on to the actual PGN).

I think if all that holds every move, there should be no way to cheat. Let me know if I'm missing anything.

[ghost]

I'm still not a fan of how we're handling challenges. Why do we need a completely separate object for it?

Why not do something like this if I start as white:

"value": {
    "is_active": true,
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
        "origin_post_id": "123"
    },
    "pgn": "e4"
}

Then, if you choose to play, you could respond with:

"value": {
    "is_active": true,
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
        "origin_post_id": "123"
    },
    "pgn": "e4  e5"
}

If I wanted to start as black, my challenge just wouldn't include a move:

"value": {
    "is_active": true,
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
        "origin_post_id": "123"
    },
    "pgn": ""
}

And, should you choose to play, you would just respond with your first move:

"value": {
    "is_active": true,
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
        "origin_post_id": "123"
    },
    "pgn": "e4"
}

With the challenge object, forking games becomes considerably more difficult. If we forego that object, one could simply grab any post from any game, and start playing. Clients can figure out who's move it is just by using some modulo arithmetic.

[ghost]

Clients can't just grab the first response, and ignore subsequent moves, either. What if the first response was an invalid move (made by a bad client)? They should grab the first valid response. But that's not that important for this discussion.

[ghost]

Also, if clients check the PGN that they receive in response with the one they sent on the previous ply, cheating should be impossible.

[ghost]

Also, if we want to be able to make moves and see what the community has to say about it, without committing to that move, we could do something like

"value": {
    "is_active": true,
    "is_final": false,
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
        "origin_post_id": "123"
    },
    "pgn": "e4"
}

If the is_final flag is set to true or is absent, that move is final and set in stone. If it's set to false, then the player isn't committed to it yet.

This is another case where the client can't just take the first response and ignore the rest.

[ghost]

I don't like the selected_ply designation for this, because clients will have to do something special to show the board. If we include the move in the pgn field, displaying the move becomes trivial.

[ghost]

If someone wants to reject a challenge, they could simply respond with a post setting is_active to false:

"value": {
    "is_active": true,
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
        "origin_post_id": "123"
    },
    "pgn": "e4"
}

Followed by:

"value": {
    "is_active": false,
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
        "origin_post_id": "123"
    },
    "pgn": "e4"
}

[jnovack]

I don't believe the structure in #issuecomment-8339017 hinders forking. Upon a successful fork, the origin_post_id should be (read IEEE "MUST BE") updated. If the fork is by a different user, both the origin_post_id and white/black names would be changed. The prior play would already be in the pgn list.

Agree with first VALID response. If we wanted to fork the game and play a second (or third) move, the origin_post_id should be (must be) updated as a normal fork.

I disagree on rejection of challenges with the is_active. Why not a "result" field which is one of "white wins", "black wins", "draw", "rejected", "ongoing"

[mattflaschen]

@davidkrauser:

You're right. I think your way of starting a challenge is better for the most part.

A trivial quibble is that you're missing move numbers. Also, I would prefer that the pgn is null when you want to be black, not an empty string; I don't know if an empty string is valid PGN.

It might be a moot point since I think that app.net also drops empty strings (https://alpha-api.app.net/stream/0/posts/312689?include_annotations=1&access_token=TOKENHERE). That was sent with pgn as an empty string, since po left it out. I don't know what happens to null.

We also need to clarify what origin_post_id is. I also don't think you can send that in a challenge. If you did, a challenge couldn't start a thread, which would be very common.

I agree that only the second or later (otherwise) valid post should be dropped.

You're right that the actual PGN needs to be checked against the prior ply; I didn't think of that. Otherwise, they can post a whole different game with the same ply count.

I like supporting is_final. Of course, you can easily see your opponent used it. People might want to ban collaboration for some games. However, that's a social issue to be resolved between the players. If we didn't have is_final, people could still post games disconnected from the thread and ask for feedback.

selected_ply is not really for correspondence chess. The main use case is for posting an old game and commenting on a particular move (selected_ply lets the board open to that move). All the moves will still be in the PGN in any event.

[mattflaschen]

@jnovack Yes, David was saying my schema hindered forking somewhat. I agree that his works better; we just have to resolve a couple points regarding origin_post_id (see above).

[mattflaschen]

I forgot. I'm envisioning correspondence chess being based on the mentions stream. So in addition to the challenges, all of the plies should also be annotated mentions.

[jnovack]

That was my fault for editing while other people were replying, I think my "result" state was overlooked.

Is there a way in the API to GET your current post number (or some self-referencing variable?) Otherwise, you can do an single "jnovack has challenged you to a dual!" and send that post ID as the original_post_id of the challenge.

As far as the check goes, you would need to check against both the parent and grand-parent. If I was cheating, a quick against MY parent (the grandparent, your last move) would solve that. So all you need is a "parent" and "grandparent" numbers.

[jnovack]

The "parent" and "grandparent" posts should also prevent against someone cheating by "fast-forwarding" the game. If we are on move 5, and I want to put you at a (hypothetical) move 48 where I trap your queen, the p/gp checking would confirm that.

Rejections, Wins and Forfeits would all have to be verified in this fashion. I should not be able to send you a "result: white wins" in the middle of the game without some sort of confirmation from black or from a thirdparty (@ deepblue)

Obviously, "black forfeits" should only be able to be sent from black's account. But this is for the client to do, my original comments are only meant to include result, parent and grandparent

[mattflaschen]

@jnovack wrote:

I disagree on rejection of challenges with the is_active. Why not a "result" field which is one of "white wins", "black wins", "draw", "rejected", "ongoing"

Yeah, I can see how this makes it a little hard to distinguish between a complete correspondence game and a rejected challenge.

One thing I keep thinking about is how much we should repeat (in JSON) information that can be encoded in the PGN directly.

I think I'm okay with those bare-bones results, even though some repeat the PGN Result tag (good clients can post both). Do we want to copy their values, "1-0", "0-1", and "1/2-1/2", though?

I think we should leave the detailed outcomes for the Termination tag, though.

I don't think a post can reliably include its own post number. I think we can use @davidkrauser's challenge format, have the first challenge post simply omit origin_post_id, and have it be that challenge post's id later. If we go this route, challenge_post_id would be clearer.

I agree about the grandparent and parent, both of which can be got easily by following two reply_to's (no need to explicitly write them). I think David and I expressed a similar concept, though not with the same terms. E.g. in my post above, B (the parent) is the post you're validating, A is the grand-parent, and you will make what we'll call C once your client proves the other player didn't cheat. :)

We can't stop players from sending bogus messages of all sorts, but it won't be displayed as a real win to the loser (or eventually observers) unless it validates. That goes for resignations too.

I think I addressed the fast forward attack in that same validation post, as well as double moves.

[jnovack]

@mattflaschen Then great, we are all on the same page. :)

Agree on challenge_post_id vs origin. Agree on termination tag.

I think this solidifies the v1.0 spec. Let's try it in practice and see where the problems are.

[mattflaschen]

Alright, let's use "1-0", "0-1", and "1/2-1/2" for the result field, for wins and draws, and rejected and ongoing otherwise.

[ghost]

Sounds good to me. Let's give it a whirl!

[mattflaschen]

I didn't think about this before, but should result be at the outer level, or in correspondence? If we put it on the outside, are we expecting clients to populate it for non-correspondence games (it's a little extra work, but as mentioned, in good PGNs, posting clients can get it from the tag header)? But it would make filtering easier, so it's likely worthwhile.

I assume that if we're using result for all games we should remove is_active.

[jnovack]

At that point, why not have everything be on the same level? Why are we breaking up everything into different levels, what purpose does it serve?

[ghost]

After thinking about it, I personally like the idea of having everything on the same level, and having a seperate "result" field like @jnovack proposed.

Something like:

"value": {
    "result": "rejected",
    "white": "davidkrauser",
    "black": "mattflaschen",
    "origin_post_id": "123"
    "pgn": "1. e4 ..."
}

I also like the set of outcomes @jnovack suggests with some minor tweaks: white, black, draw, active, and rejected

[jnovack]

If we are keeping the pgn field as standard, let's also keep the result field as standard and utilize the termination tag, again, as standard, with the addition of "rejected". Under one heading, that keeps everything nice and neat, and, well, standard. ;)

I like the idea of challenge_post_id being the annotation added upon acceptance or rejected.

"value": {
    "white": "davidkrauser",
    "black": "mattflaschen",
    "pgn": "1. e4 e5",
    "parent" : null
}

then (parent moves to grandparent, challenge_post_id added to first post, pgn updated, result added to indicate ongoing)

"value": {
    "white": "davidkrauser",
    "black": "mattflaschen",
    "challenge_post_id": "123",
    "pgn": "1.e4 e5 2.Nf3 d6",
    "parent": "123",
    "result" : "*",
    "grandparent" : null
}

finally (last move of the game -- pgn continues updating, parent/grandparent rolling appropriately, termination added, result updated)

"value": {
    "white": "davidkrauser",
    "black": "mattflaschen",
    "challenge_post_id": "123",
    "pgn": "1.e4 e5 2.Nf3 d6 ...",
    "parent": "567",
    "grandparent" : "534",
    "result": "1-0",
    "termination": "normal"
}

I feel that the losing party (or winning party in a forfeit) should REPEAT the last post. This is a "confirmation" that the last move wasn't cheating and the game is officially done. Clients should do this automatically.

A rejection should be as follows:

"value": {
    "white": "davidkrauser",
    "black": "mattflaschen",
    "challenge_post_id": "123",
    "pgn": "1.e4 e5",
    "parent": "123",
    "grandparent" : null,
    "result" : "draw",
    "termination" : "rejected"
}

During a natural game, a valid client (using IEEE terminology): MUST validate parents and grandparents came from white/black as appropriate MUST validate parents and grandparents having proper pgn notation MUST validate the last move was valid (for the piece) MUST confirm challenge_post_id SHOULD repeat the last post

Upon a fork, a valid client: MUST preform the above, MUST change black/white as appropriate MUST remove challenge_post_id MUST remove parent/grandparents (in essense, they are creating a NEW game with the PGN partially filled in) (if we want to, we can create a fork_from_id with the last original parent)

[mattflaschen]

Remember, this is not only for correspondence chess. It's also for sharing historical games (which is already implemented). The purpose of the different levels is two-fold:

1. It's an easy way to see that it's a correspondence game (check for the property existing).
2. It's the place to put values that only make sense for correspondence. I can't put Bobby Fischer's app.net username in correspondence.black (let alone challenge_post_id), but that's okay since I left out correspondence entirely.

I'm fine with putting result on the top-level, but I would like to use the values from the PGN result tag where possible, so they can be copied in for historical games.

I'm okay with using * for ongoing (copying the PGN standard) and leaving it out for challenges (no game/no result), but I would like to use rejected in the result as we said earlier. A rejected challenge is not a draw.

I thought we weren't going to repeat Termination in the JSON, because those details weren't essential and could be parsed from the PGN.

As I said before, parent and grandparent should not be written. They can be calculated easily from reply_to (everything should be a reply with a mention) by the viewer, so there's no need to add another field to be checked for forgery.

I think is_final only makes sense for correspondence, so that should be moved in:

So I propose:

Non-correspondence where white won (... would not be in real PGN) with optional selected_ply:

"value": {
    "result": "1-0",
    "pgn": "[Date \"2011.07.22\"] ... 1. e4 e5 ... 33.Qxe7# 1-0",
    "selected_ply": "5"
}

Challenge as white:

"value": {
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
    },
    "pgn": "1. e4"
}

Challenge as black:

"value": {
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
    }
}

Black accepts:

"value": {
    "result": "*",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123"
    },
    "pgn": "1. e4 e5",
}

Black rejects:

"value": {
    "result": "rejected",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123"
    },
    "pgn": "1. e4"
}

White accepts:

"value": {
    "result": "*",
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
         "challenge_post_id": "123"
    },
    "pgn": "1. d4",
}

White rejects:

"value": {
    "result": "rejected",
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
        "challenge_post_id": "123"
    }
}

Follow-up black move after white acceptance:

"value": {
    "result": "*",
    "correspondence": {
        "white": "davidkrauser",
        "black": "mattflaschen",
         "challenge_post_id": "123"
    },
    "pgn": "1. d4 d5",
}

Follow-up white move after black acceptance:

"value": {
    "result": "*",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123"
    },
    "pgn": "1. e4 e5 2. d4",
}

Hypothetical black move:

"value": {
    "result": "*",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123",
         "is_final": false
    },
    "pgn": "1. e4 e5 2. d4 d5",
}

White wins by checkmate:

"value": {
    "result": "1-0",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123"
    },
    "pgn": "1. e4 e5 2. d4 ... 23. Be7# 1-0",
}

White resigns/black wins:

"value": {
    "result": "0-1",
    "correspondence": {
        "white": "mattflaschen",
        "black": "davidkrauser",
         "challenge_post_id": "123"
    },
    "pgn": "[Termination \"davidkrauser won by resignation\"] 1. e4 e5 2. d4 ... 23. 0-1",
}

I think I agree with all of your shoulds and musts except the parent/grandparent. I think my validation above still applies (except that it's based on challenge_post_id now.

We should get this in the wiki once we're okay with it.

[ghost]

I think that looks good. Let's put this up on the wiki and solidify version 1.0 of the spec.

Also, at this point I think we should we put the annotations in the net.app.games.chess namespace.

[mattflaschen]

I think it should be net.app.core.games.chess but per the docs we need approval, so let's keep using the temporary one for now.

I've copied my last post to the wiki (Chess Annotation), but I'll leave a little more time for feedback before I update my code.

[mattflaschen]

Alright, for the record, I propose games.chess as the annotation type, per berg's feedback.

[mattflaschen]

What do people think about adding "version": "1.0" at the root of all values. Alternatively, would it be better to add .v1 to the namespace? This idea is based on a suggestion by Josh Blake.

[ghost]

Informative conversation on ADN about namespace and spec: https://alpha.app.net/davidkrauser/post/360384

[ghost]

I like having "version" in the root. Let's add that to the spec.

[mattflaschen]

Alright, I'm also going to change it to be games.chess.

[mattflaschen]

We need a way to offer draws by agreement. How about, when you make your move, you can include

"offer_draw": true

in the correspondence object. The opponent would not include that field when replying.

To accept, they would use the 1/2-1/2 results, and ideally the Termination tag, as discussed. To reject, they just make their move as normal.

[jnovack]

Other than creating another tag (which I don't see a way around), it looks good. Is this to combat "never-ending" games because neither person can win?

[ghost]

Sounds good to me, too.

Would make games with only kings and pawns left easier to end. On Sep 23, 2012 3:41 PM, "jnovack" notifications@github.com wrote:

Other than creating another tag (which I don't see a way around), it looks good. Is this to combat "never-ending" games because neither person can win?

— Reply to this email directly or view it on GitHubhttps://github.com/appdotnet/api-spec/issues/164#issuecomment-8801982.

[mattflaschen]

@jnovack the two people can agree to a draw for any reason. However, etiquette is to only do it when you think it will end in a draw anyway (e.g. due to insufficient material or 50-move rule).

Alright, I'll probably implement this soon.

[mattflaschen]

Just to follow up, this is up at http://dev.apppassant.com/ . See https://github.com/mattflaschen/apppassant/issues for what is still being worked on in App Passant.

Of course other implementations (read-only or read-write) are welcome.

I'm closing this for now.

[mattflaschen]

I'm going to reimplement this using the channel API. I think it's a pretty good fit.

Besides the obvious (private games without cryptography, native security (less constantly checking for shenanigans), a simple way of defining players (writers) and spectators (readers who are not writers)), it allows us to get "unread" (it's my turn) moves for all games easily. Using our own channel type should avoid regular clients messing with the channel-specific stream marker (which is the main reason why I didn't use it before).

My question is:

Is anyone else planning on working on this (chess on app.net)?

If so, I'll gladly do the whole standardization process again. But otherwise, I'll just go ahead and implement, then document after the fact.

The PGN part will probably remain the same, as will the basic order of operations. But I'll probably e.g. use channel_id in place of challenge_post_id, and various related changes.

[orianmarx]

@mattflaschen I'm closing this issue. If you'd like it to remain active, please open a new issue on https://github.com/appdotnet/object-metadata and include a reference to appdotnet/api-spec#164 to link back to this discussion.

[mattflaschen]

See https://github.com/appdotnet/object-metadata/issues/12