hdinsight / hdinsight-storm-examples

This is a repository for complete and easy to use samples that demonstrate the use of Apache Storm on HDInsight
Apache License 2.0
59 stars 52 forks source link

Sample code that writes to an Azure Blob #1

Closed dvmorris closed 9 years ago

dvmorris commented 9 years ago

Do you have any sample code where Storm writes out to an Azure blob, rather than HBase? We are having trouble getting any examples to work that do not write to HBase.

shanyu commented 9 years ago

Hi Dave,

Yes, the IotSample shows how to write to Azure Blob Storage using the storm-hdfs project.

Thanks, Shanyu

From: Dave Morris [mailto:notifications@github.com] Sent: Monday, February 16, 2015 6:29 AM To: hdinsight/hdinsight-storm-examples Subject: [hdinsight-storm-examples] Sample code that writes to an Azure Blob (#1)

Do you have any sample code where Storm writes out to an Azure blob, rather than HBase? We are having trouble getting any examples to work that do not write to HBase.

— Reply to this email directly or view it on GitHubhttps://github.com/hdinsight/hdinsight-storm-examples/issues/1.

dvmorris commented 9 years ago

Could you send me a URL to that sample code? I cannot find it on GitHub.

shanyu commented 9 years ago

The code will be pushed out in a day or so. I appreciate your patience.

Thanks, Shanyu

From: Dave Morris [mailto:notifications@github.com] Sent: Monday, February 16, 2015 2:32 PM To: hdinsight/hdinsight-storm-examples Cc: Shanyu Zhao Subject: Re: [hdinsight-storm-examples] Sample code that writes to an Azure Blob (#1)

Could you send me a URL to that sample code? I cannot find it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/hdinsight/hdinsight-storm-examples/issues/1#issuecomment-74582654.

dvmorris commented 9 years ago

Sounds good. Looking forward to seeing the code!

On Mon, Feb 16, 2015 at 4:54 PM, Shanyu Zhao notifications@github.com wrote:

The code will be pushed out in a day or so. I appreciate your patience.

Thanks, Shanyu

From: Dave Morris [mailto:notifications@github.com] Sent: Monday, February 16, 2015 2:32 PM To: hdinsight/hdinsight-storm-examples Cc: Shanyu Zhao Subject: Re: [hdinsight-storm-examples] Sample code that writes to an Azure Blob (#1)

Could you send me a URL to that sample code? I cannot find it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/hdinsight/hdinsight-storm-examples/issues/1#issuecomment-74582654>.

— Reply to this email directly or view it on GitHub https://github.com/hdinsight/hdinsight-storm-examples/issues/1#issuecomment-74585189 .

ravitandonrt commented 9 years ago

Please take a look at: IotTopology.java in IotExample

https://github.com/hdinsight/hdinsight-storm-examples/blob/master/IotExample/iot/src/main/java/com/microsoft/hdinsight/storm/examples/IotTopology.java

    //Create HdfsBolt to store data to Windows Azure Blob Storage
    SyncPolicy syncPolicy = new CountSyncPolicy(1000);
    FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(1.0f, Units.MB);

    RecordFormat recordFormat = new DelimitedRecordFormat().withFieldDelimiter(",");
    FileNameFormat fileNameFormat = new DefaultFileNameFormat().withPath("/iotexampledata/");
    HdfsBolt wasbBolt = new HdfsBolt()
      .withFsUrl("wasb:///")
      .withRecordFormat(recordFormat)
      .withFileNameFormat(fileNameFormat)
      .withRotationPolicy(rotationPolicy)
      .withSyncPolicy(syncPolicy);

    topologyBuilder.setBolt("WasbStoreBolt", wasbBolt, 10)
      .fieldsGrouping("DataReferenceBolt", new Fields("model")).setNumTasks(10);