search

HazyResearch / bazaar

14 stars 11 forks source link

Condor/AWS unified interface? #1

Open chrismre opened 9 years ago

chrismre commented 9 years ago

This is awesome, and I want to re-parse all of open data with it.

Can we make sure that we can run easily on AWS and Condor? Let's just make sure there are simple scripts for us to make it easy.

Ideally, we could make Bazaar a submodule of DeepDive that makes it super easy to parse users own documents (locally, and then an AWS directory, and a condor directory, whatever...)

@zhangce @netj @raphaelhoffmann @feiranwang @SenWu @alldefector

Awesome work, Raphael!

chrismre commented 9 years ago

Also, Azure should be supported--since we plan to run there (and whatever else is on MEMEX?)

raphaelhoffmann commented 9 years ago

Azure is now fully supported.

Condor is using its own tools and formats for job submission. We can't use "fabric" here the way we use if for ec-2 and Azure. We may want a wrapper for Condor that exposes only a simple interface for submitting nlp pipeline jobs.