Lab 1: MapReduce

[moyuanhuang]

• [x] Part 1: Map/Reduce input and output

[moyuanhuang]

• [x] Part 2: Single-worker word count

[moyuanhuang]

• [x] Part 3: Distributing MapReduce tasks

这个部分主要是用多线程模拟分布式的MapReduce。包含了test_test.go中的两个测试用例。虽然最后只有几十行代码，但涉及了这次实验中的很多文件以及go中的并行机制。

$ go test -run  TestParallelBasic
...
/var/tmp/824-501/mr20072-master: Map/Reduce task completed
PASS
ok      mapreduce   5.364s
$ go test -run TestParallelCheck
...
/var/tmp/824-501/mr19543-master: Map/Reduce task completed
PASS
ok      mapreduce   15.080s

对函数及变量的理解

• call() in common_rpc.go
• newCond *sync.Cond: member of type Master struct
• registerCh chan string in schedule.go register() Register() forwardRegistrations()

go的并发鸡翅机制

• sync.mutex
• sync.waitGroup
• sync.Cond

[moyuanhuang]

• [x] Part 4: Handling worker failures

这次试验写到这里对整个代码的框架已经比较熟悉了，加上第四部分也比较简单，很快就能写完。我的做法是在schedule()中加入一个taskCh chan string来传入所有待处理的任务（包括没有处理过以及处理失败的）。对于每一个任务，如果RPC失败就重新把任务发送到taskCh中。

schedule()不断从管道中读取任务直到管道中没有任务，说明所有任务都被至少成功处理了一遍。这个时候通过default语句进行跳出循环的操作。

需要注意的地方是如果新建任务管道时没有生命缓存大小，向管道里输入数据是会阻塞的，因此需要使用goroutine或者声明缓存大小taskCh := make(chan string, 100)。关于channel的操作在这里有大概说明。

[moyuanhuang]

• [x] Part 5: Inverted index generation

Inverted indices are widely used in computer science, and are particularly useful in document searching. Broadly speaking, an inverted index is a map from interesting facts about the underlying data, to the original location of that data. For example, in the context of search, it might be a map from keywords to documents that contain those words.

mapF()和reduceF()可以说是leetcode easy难度了...顺便再熟悉一下go中的基本语法。

type rune // rune = int32 (see golang.org/issue/21601) fmt.Sprintf(format string, a ...interface{}) string // get a formatted string strings.Join(a []string, sep string) string // join a set of strings with sep