paulhoule
commented
10 years ago Ok, here is the command used for the first tiny successful tiny run
haruhi run job -clusterId tinyAwsCluster -jarId telepath projectNormalized3D -input s3n://wikimedia-pagecounts/2008/2008-01/pagecounts-20080101-000000.gz -factors s3n://wikimedia-summary/monthly-sites.txt.gz -output s3n://wikimedia-summary/test/firstNormalized -R 1let's do a bigger run
haruhi run job -clusterId smallAwsCluster -jarId telepath projectNormalized3D -input s3n://wikimedia-pagecounts/2008/2008-01/ -factors s3n://wikimedia-summary/monthly-sites.txt.gz -output s3n://wikimedia-summary/test/secondNormalized -R 4
For this we divide the monthlies by the normalization factors for the project*month and then we normalize each project so that it adds up to, let's say, 5.
The issue is that these probabilities don't add up to 1 for several reasons. For one thing, people think about all kinds of things that aren't in Wikipedia, so a large amount of importance is not represented, that is, for some purposes, the probabilities might not add up to one.
An alternative issue is that often people are thinking about more than one thing at a time, you might be thinking about the relationship between A and B or you are thinking about 'Rhode Island' so you're also thinking about the `United States'. If you're thinking about "Electric Funeral" you're thinking about "Black Sabbath" and "Ozzy Osborne", etc.
I get 5 from the idea that the human cognitive capacity is
http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two
The moral of it is that if you want to use this data as a prior distribution you'll need to tune the normalization for your data set.