Open ysh329 opened 5 years ago
text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")
Speed Run workloads 100x faster.
Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.
#include <stdio.h>
//函数指针申明
typedef int (*mapFunction)(int);
typedef int (*reduceFunction)(int,int);
#define ERROR -1;
//-----------------Map和Reduce操作-----------------
/*
* 对list数组的数据进行处理,然后存储在list数组中
*/
void map(mapFunction func,int *list,int len)
{
int i;
for(i=0;i<len;i++)
{
list[i] = func(list[i]);
}
}
int reduce(reduceFunction func,int *list,int len)
{
if(len <= 0) return ERROR;
int retVal = 0;
for(int i=0; i<len; i++){
retVal = func(retVal, list[i]);
return retVal;
}
//-----------------------测试-------------------------
int square(int i) {
return i*i;
}
int add(int i,int j) {
return i+j;
}
int main(int argc,char*argv[]) {
int array[5];
for(int i=0; i<5; i++){
array[i] = i;
}
mapFunction mapFuncPointer = (mapFunction)□
reduceFunction reduceFuncPointer = (reduceFunction)&add;
map(mapFuncPointer,array,5);
int result = reduce(reduceFuncPointer,array,5);
printf("The result is %d ",result);
return 0;
}
作者:naivebaby 来源:CSDN 原文:https://blog.csdn.net/naivebaby/article/details/1357457 版权声明:本文为博主原创文章,转载请附上博文链接!
算法数据结构