Randomly select a subset of attributes for each scenario

[anushabala]

@hhexiy @percyliang @mihail911 He, this code still doesn't work - I get a IndexError: list index out of range on line 68 of generate_scenarios.py, and I'm not sure why that is..

[hhexiy]

Also we need to change the schema for the KB to match the subset of attributes, right? Currently there's an error when dumping the KB because the schema is not updated.

[anushabala]

@hhexiy good point about changing the schema.. I'll update the code so that each KB has a different associated schema - do you think that's the best way to do it?

[hhexiy]

Hmm, actually the KB just needs (ordered) attributes in the schema. What do you think of associating each KB with a list attributes instead of a schema? Would that be easier?

On Sun, Nov 20, 2016 at 10:48 AM, anushabala notifications@github.com wrote:

@hhexiy https://github.com/hhexiy good point about changing the schema.. I'll update the code so that each KB has a different associated schema - do you think that's the best way to do it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-261796672, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJYpv7YD_L4Our7900dZ9pjkUgQvrSCks5rAJX-gaJpZM4K2-ZH .

[anushabala]

Hmm I think that might be easier! On Sun, Nov 20, 2016 at 9:15 PM hhexiy notifications@github.com wrote:

Hmm, actually the KB just needs (ordered) attributes in the schema. What do you think of associating each KB with a list attributes instead of a schema? Would that be easier?

On Sun, Nov 20, 2016 at 10:48 AM, anushabala notifications@github.com wrote:

@hhexiy https://github.com/hhexiy good point about changing the schema.. I'll update the code so that each KB has a different associated schema - do you think that's the best way to do it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-261796672 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ABJYpv7YD_L4Our7900dZ9pjkUgQvrSCks5rAJX-gaJpZM4K2-ZH

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-261845512, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLcv_hKCayaSSGYIlW5sELaT05df57vks5rASkEgaJpZM4K2-ZH .

Anusha Balakrishnan M.S. Computer Science (Artificial Intelligence) Stanford University '17 anusha@cs.stanford.edu

[percyliang]

So what exactly will the JSON for each scenario look like?

[anushabala]

@percyliang I don't think the scenario representation will change much here, except maybe that each KB in the scenario will also have a list of the attributes present in that KB (so that we know what attributes to load when loading the scenarios from file). @hhexiy does that sound right?

[hhexiy]

so that we know what attributes to load when loading the scenarios from file

Not sure I understand this. I thought we're generating each scenarios from scratch, i.e. first select a subset of attributes, then fill in the values. Here it sounds like it's loading a sub-scenario given a complete scenario??

My understanding was that:

• We first generate a set of scenarios with different subset of attributes, they are written to a json file and are loaded during dialogue collection/generation.

• When we dumping the dialogues each example also has a 'scenario' field (as in the current PR).

• The KB dict in the JSON file is the same as before: a list of items.

• The KB object has an 'attributes' field which replace the 'schema' field because KBs have different schema now. This is only used for displaying a KB (headers etc.).

So re @percyliang 's questions, the JSON for each scenarios does not change.

[percyliang]

On Fri, Nov 25, 2016 at 2:26 PM, hhexiy notifications@github.com wrote:

so that we know what attributes to load when loading the scenarios from file Not sure I understand this. I thought we're generating each scenarios from scratch, i.e. first select a subset of attributes, then fill in the values. Here it sounds like it's loading a sub-scenario given a complete scenario??

My understanding was that:

-

We first generate a set of scenarios with different subset of attributes, they are written to a json file and are loaded during dialogue collection/generation.

When we dumping the dialogues each example also has a 'scenario' field (as in the current PR).

The KB dict in the JSON file is the same as before: a list of items.

The KB object has an 'attributes' field which replace the 'schema' field because KBs have different schema now. This is only used for displaying a KB (headers etc.).

Yes, that makes sense to me. Do we even need an attributes field since can't the attributes be recovered from the items?

-

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-263028473, or mute the thread https://github.com/notifications/unsubscribe-auth/AAakuAHSna7Qswhr81AfOtHPsnFJnDNpks5rB2CwgaJpZM4K2-ZH .

[anushabala]

This makes sense! I think we probably still need the list of attributes since the items are all dictionaries and we want the attributes to be ordered. On Sun, Nov 27, 2016 at 8:00 AM Percy Liang notifications@github.com wrote:

On Fri, Nov 25, 2016 at 2:26 PM, hhexiy notifications@github.com wrote:

so that we know what attributes to load when loading the scenarios from file Not sure I understand this. I thought we're generating each scenarios from scratch, i.e. first select a subset of attributes, then fill in the values. Here it sounds like it's loading a sub-scenario given a complete scenario??

My understanding was that:

-

We first generate a set of scenarios with different subset of attributes, they are written to a json file and are loaded during dialogue collection/generation.

When we dumping the dialogues each example also has a 'scenario' field (as in the current PR).

The KB dict in the JSON file is the same as before: a list of items.

The KB object has an 'attributes' field which replace the 'schema' field because KBs have different schema now. This is only used for displaying a KB (headers etc.).

Yes, that makes sense to me. Do we even need an attributes field since can't the attributes be recovered from the items?

-

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-263028473 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AAakuAHSna7Qswhr81AfOtHPsnFJnDNpks5rB2CwgaJpZM4K2-ZH

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-263129960, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLcvy5sAmAlHJi79iyFxj5NyN-QUHxgks5rCakSgaJpZM4K2-ZH .

-- Anusha Balakrishnan M.S. Computer Science (Artificial Intelligence) Stanford University '17 anusha@cs.stanford.edu

[percyliang]

Ok, that works.

On Sun, Nov 27, 2016 at 10:10 AM, anushabala notifications@github.com wrote:

This makes sense! I think we probably still need the list of attributes since the items are all dictionaries and we want the attributes to be ordered. On Sun, Nov 27, 2016 at 8:00 AM Percy Liang notifications@github.com wrote:

On Fri, Nov 25, 2016 at 2:26 PM, hhexiy notifications@github.com wrote:

so that we know what attributes to load when loading the scenarios from file Not sure I understand this. I thought we're generating each scenarios from scratch, i.e. first select a subset of attributes, then fill in the values. Here it sounds like it's loading a sub-scenario given a complete scenario??

My understanding was that:

-

We first generate a set of scenarios with different subset of attributes, they are written to a json file and are loaded during dialogue collection/generation.

When we dumping the dialogues each example also has a 'scenario' field (as in the current PR).

The KB dict in the JSON file is the same as before: a list of items.

The KB object has an 'attributes' field which replace the 'schema' field because KBs have different schema now. This is only used for displaying a KB (headers etc.).

Yes, that makes sense to me. Do we even need an attributes field since can't the attributes be recovered from the items?

-

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/stanfordnlp/game-dialogue/ pull/17#issuecomment-263028473 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ AAakuAHSna7Qswhr81AfOtHPsnFJnDNpks5rB2CwgaJpZM4K2-ZH

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/ pull/17#issuecomment-263129960, or mute the thread https://github.com/notifications/unsubscribe-auth/ ABLcvy5sAmAlHJi79iyFxj5NyN-QUHxgks5rCakSgaJpZM4K2-ZH .

-- Anusha Balakrishnan M.S. Computer Science (Artificial Intelligence) Stanford University '17 anusha@cs.stanford.edu

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-263137120, or mute the thread https://github.com/notifications/unsubscribe-auth/AAakuNvESLM5uH5fEAxtV3C0sZfMhq9Aks5rCceFgaJpZM4K2-ZH .

[anushabala]

@hhexiy @percyliang @mihail911 This script finally works! Sample scenarios from the script are attached It selects between 3 and 5 attributes at random for each scenario (the number of attributes is also randomly picked).

scenarios.txt

Sample JSON of a scenario (the only new thing that was added is the attribute list, in bold):

{"attributes": [{"unique": false, "value_type": "school", "name": "School"}, {"unique": false, "value_type": "hobby", "name": "Hobby"}, {"uniq ue": false, "value_type": "loc_pref", "name": "Location Preference"}], "uuid": "S_VBhYn1G6iQF0rpuc", "kbs": [[{"Hobby": "Skydiving", "School": "University of Maine at Fort Kent", "Location Preference": "indoor"}, {"Hobby": "Creative writing", "School": "University of Maine at Fort Kent ", "Location Preference": "indoor"}, {"Hobby": "Computer programming", "School": "Purdue University Calumet", "Location Preference": "indoor"}, {"Hobby": "Computer programming", "School": "University of Wisconsin-Platteville", "Location Preference": "outdoor"}, {"Hobby": "Flying disc", "School": "Humboldt State University", "Location Preference": "indoor"}, {"Hobby": "Sculling", "School": "Humboldt State University", "Locatio n Preference": "indoor"}, {"Hobby": "Scouting", "School": "Oregon Health Sciences University", "Location Preference": "outdoor"}, {"Hobby": "Co mputer programming", "School": "University of Maine at Fort Kent", "Location Preference": "indoor"}, {"Hobby": "Sculling", "School": "The Unifo rmed Services University of the Health Sciences", "Location Preference": "outdoor"}, {"Hobby": "Do it yourself", "School": "Humboldt State Univ ersity", "Location Preference": "indoor"}], [{"Hobby": "Baton Twirling", "School": "Mount Senario College", "Location Preference": "outdoor"}, {"Hobby": "Kabaddi", "School": "Concordia College-Ann Arbor", "Location Preference": "outdoor"}, {"Hobby": "Baton Twirling", "School": "Purdue University Calumet", "Location Preference": "outdoor"}, {"Hobby": "Creative writing", "School": "Mount Senario College", "Location Preference": "outdoor"}, {"Hobby": "Aqua-lung", "School": "University of Maine at Fort Kent", "Location Preference": "indoor"}, {"Hobby": "Rowing", "School ": "University of Maine at Fort Kent", "Location Preference": "indoor"}, {"Hobby": "Rowing", "School": "University of Maine at Fort Kent", "Loc ation Preference": "outdoor"}, {"Hobby": "Baton Twirling", "School": "University of Chicago", "Location Preference": "indoor"}, {"Hobby": "Rowi ng", "School": "Antioch University-Seattle", "Location Preference": "indoor"}, {"Hobby": "Creative writing", "School": "University of Maine at Fort Kent", "Location Preference": "indoor"}]]}

[hhexiy]

This looks good! Thanks!

One question: how are the alphas generated now? We could have a set of alphas for each attribute too (simplest set would be 1 and 0.1 for example) and randomly select from them. We could also just fix the alphas for each attribute, since we are randomizing over the attributes we'll have different distributions.

Also, shall we randomize the number of items too, say 5-10?

[anushabala]

Right now the alphas are set like this:

if domain == 'Matchmaking': self.alphas = [1.] * len(self.attributes) else: alphas = list(np.linspace(1, 0.1, len(self.attributes))) self.alphas = dict((attr, alpha) for (attr, alpha) in izip(self.attributes, alphas)) for i, attr in enumerate(self.attributes): if attr.name == 'Name': self.alphas[attr] = 2 elif attr.name == 'Hobby': self.alphas[attr] = 0.5 elif attr.name == 'Time Preference': self.alphas[attr] = 1.0 elif attr.name == 'Location Preference': self.alphas[attr] = 1.0

So I guess right now the alphas for a fair number of attributes are just fixed (I did this just to preserve the old alpha values you were using). I agree that it should be randomized, maybe each attribute can have an alpha from the set list(np.linspace(1, 0.1, len(self.attributes))) ? Except for Name and Hobby (and maybe time and location preference?) - were the alphas hardcoded to a specific value there for a reason?

And yeah, randomizing the number of items is a good idea!! I like the idea of choosing in the range [5,10], I can change it to that.

[hhexiy]

Except for Name and Hobby (and maybe time and location preference?) - were the alphas hardcoded to a specific value there for a reason?

Nothing important. I think I fixed alphas for these attributes because time and location only have a few values and I wanted it to be more uniform, otherwise it often has a single value. You can ignore it.

Choosing alphas from the linspace is a good idea! Do you mean randomize alpha for each attribute and each scenario? If so, we probably also need to log the alphas in case later we want to group the scenarios based on the alphas. Alternatively, we can set alphas here and fix it for all scenarios -- then there isn't much difference between random selection and assignment (as long as we don't assign the same alpha to all attributes). But if we sampled all "uniform" attributes for a scenario that's not good... So maybe we need to sample an alpha for each attribute for each scenario, and also make sure at least one attribute has a skewed distribution??

On Wed, Nov 30, 2016 at 4:52 PM, anushabala notifications@github.com wrote:

Right now the alphas are set like this:

if domain == 'Matchmaking': self.alphas = [1.] * len(self.attributes) else: alphas = list(np.linspace(1, 0.1, len(self.attributes))) self.alphas = dict((attr, alpha) for (attr, alpha) in izip(self.attributes, alphas)) for i, attr in enumerate(self.attributes): if attr.name == 'Name': self.alphas[attr] = 2 elif attr.name == 'Hobby': self.alphas[attr] = 0.5 elif attr.name == 'Time Preference': self.alphas[attr] = 1.0 elif attr.name == 'Location Preference': self.alphas[attr] = 1.0

So I guess right now the alphas for a fair number of attributes are just fixed (I did this just to preserve the old alpha values you were using). I agree that it should be randomized, maybe each attribute can have an alpha from the set list(np.linspace(1, 0.1, len(self.attributes))) ? Except for Name and Hobby (and maybe time and location preference?) - were the alphas hardcoded to a specific value there for a reason?

And yeah, randomizing the number of items is a good idea!! I like the idea of choosing in the range [5,10], I can change it to that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-264045738, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJYptBV6l6OTnh9YZHR-e__uYtCeMHTks5rDhpqgaJpZM4K2-ZH .

[percyliang]

Recording the alphas is definitely a good idea. Does it work to just draw alphas from a fixed set like {0.1, 0.5, 0.9}? It might be easier to think about the attributes as falling into qualitative groups? When you manage to generate some data, could you upload to CodaLab and send me a link? Thanks!

On Wed, Nov 30, 2016 at 7:47 PM, hhexiy notifications@github.com wrote:

Except for Name and Hobby (and maybe time and location preference?) - were the alphas hardcoded to a specific value there for a reason?

Nothing important. I think I fixed alphas for these attributes because time and location only have a few values and I wanted it to be more uniform, otherwise it often has a single value. You can ignore it.

Choosing alphas from the linspace is a good idea! Do you mean randomize alpha for each attribute and each scenario? If so, we probably also need to log the alphas in case later we want to group the scenarios based on the alphas. Alternatively, we can set alphas here and fix it for all scenarios -- then there isn't much difference between random selection and assignment (as long as we don't assign the same alpha to all attributes). But if we sampled all "uniform" attributes for a scenario that's not good... So maybe we need to sample an alpha for each attribute for each scenario, and also make sure at least one attribute has a skewed distribution??

On Wed, Nov 30, 2016 at 4:52 PM, anushabala notifications@github.com wrote:

Right now the alphas are set like this:

if domain == 'Matchmaking': self.alphas = [1.] * len(self.attributes) else: alphas = list(np.linspace(1, 0.1, len(self.attributes))) self.alphas = dict((attr, alpha) for (attr, alpha) in izip(self.attributes, alphas)) for i, attr in enumerate(self.attributes): if attr.name == 'Name': self.alphas[attr] = 2 elif attr.name == 'Hobby': self.alphas[attr] = 0.5 elif attr.name == 'Time Preference': self.alphas[attr] = 1.0 elif attr.name == 'Location Preference': self.alphas[attr] = 1.0

So I guess right now the alphas for a fair number of attributes are just fixed (I did this just to preserve the old alpha values you were using). I agree that it should be randomized, maybe each attribute can have an alpha from the set list(np.linspace(1, 0.1, len(self.attributes))) ? Except for Name and Hobby (and maybe time and location preference?) - were the alphas hardcoded to a specific value there for a reason?

And yeah, randomizing the number of items is a good idea!! I like the idea of choosing in the range [5,10], I can change it to that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/ pull/17#issuecomment-264045738, or mute the thread https://github.com/notifications/unsubscribe- auth/ABJYptBV6l6OTnh9YZHR-e__uYtCeMHTks5rDhpqgaJpZM4K2-ZH .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-264071458, or mute the thread https://github.com/notifications/unsubscribe-auth/AAakuAQk5tk4BKrFr5lEKiHXSpMJgKvfks5rDkNKgaJpZM4K2-ZH .

[anushabala]

@hhexiy Yeah, that's what I meant! So for each attribute and scenario we would pick a value from {0.1, 0.2,.... 1.0} (@percyliang I think that's what you meant too?) And yeah logging them sounds like a good idea. Since I'm already adding the attributes to the scenario maybe I can just add the alpha value for each attribute to the JSON dict for that attribute? e.g. in the example above the attributes list would look like "attributes": [{"unique": false, "value_type": "school", "name": "School", "alpha":0.5}, {"unique": false, "value_type": "hobby", "name": "Hobby", "alpha":0.9}, {"unique": false, "value_type": "loc_pref", "name": "Location Preference", "alpha":0.6}]

[percyliang]

I was suggesting that we focus our alphas only on a selected set of values that are qualitatively distinguishable (e.g., {0.1, 0.5, 1.0}) rather than allowing a lot of possibilities. I feel like this makes it easier to understand what each scenario is "about" and also in this case shifts the distribution more towards extreme alphas, which might be good (note completely sure). I'd just generate a large set of scenarios and tweak things so that when you look at scenarios, they look diverse/interesting.

On Thu, Dec 1, 2016 at 7:34 AM, anushabala notifications@github.com wrote:

@hhexiy https://github.com/hhexiy Yeah, that's what I meant! So for each attribute and scenario we would pick a value from {0.1, 0.2,.... 1.0} ( @percyliang https://github.com/percyliang I think that's what you meant too?) And yeah logging them sounds like a good idea. Since I'm already adding the attributes to the scenario maybe I can just add the alpha value for each attribute to the JSON dict for that attribute? e.g. in the example above the attributes list would look like "attributes": [{"unique": false, "value_type": "school", "name": "School", "alpha":0.5}, {"unique": false, "value_type": "hobby", "name": "Hobby", "alpha":0.9}, {"unique": false, "value_type": "loc_pref", "name": "Location Preference", "alpha":0.6}]

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/game-dialogue/pull/17#issuecomment-264204836, or mute the thread https://github.com/notifications/unsubscribe-auth/AAakuKYyzQ4ubhOG3aONbyU2CHiuPEgdks5rDukEgaJpZM4K2-ZH .

[anushabala]

Oh I see what you mean! Yeah I like the idea of choosing from that set - let me experiment with that and send over a Codalab link so we can see what the data looks like!