DATAVIEW is a big data workflow management system. It uses Dropbox as the data cloud and Amazon EC2 as the compute cloud. Current research focuses on the security and privacy aspects of DATAVIEW as well as performance and cost optimization for running workflows in clouds.
1) Release day: to be discussed --- 07/01/19
2) Video: how to create access keys -- Done
3) Integrate with Ishtiaq's workflow executor alpha and workflow planner E2C2D (not SGX-E2C2D) --Done
4) Implement the Dropbox component so that DATAVIEW does not rely on another external script. (depends on a PHP that is outside of DATAVIEW). (ongoing work)
5) Use provenance to modify the configuration files for workflow scheduling algorithm LPOD. (ongoing work)
6) Deploy DATAVIEW as a web site. (Done)
7) We will use json format for config.json file, which is the file that configures the whole
DATAVIEW system, the following properties of the system are considered:
a) WorkflowInputDir_DropboxFileSystem=DATAVIEW_INPUT
b) WorkflowInputDir_LocalFileSystem=.
c) accessKeyID and secretAccessKey, VMImageID,
d) confidentialTask = taskname
e) <VMType, IP>
The directory locations to put tasks, input data, and output data. -- ongoing work
8) Use Javadoc comments so that we can generate DATAVIEW APIs easily. ----- ongoing work
9) workflow structure visualization ---- Done
11) Provenance node miss spelling --Done
12) full name for EST ----- Done
13) In Everywhere token --> dropboxToken --Done
14) send the workflow final outputs to the corresponding files to the DATAVIEW master node. --Done
15) Introduce a data structure called taskExecutionStatus in JSON. -- Done
16) Implement the WorkflowExecutor_localBeta -- Done
17） Change to dataMove() method in taskExecutor.java --Done
18) Code refactoring: Where should these files belong? CmdLineDriver.java, MSGClient.java, MakeMachinesReady.java, MoveDataToCloud.java, MoveToCloud.java. -- ongoing work
19) Abstract classes for VMProvisioner and TaskExecutor:
What common methods should we have for VMProvisioner.java and
TaskExecutor.java? Maybe we can define an abstract VMProvisioner.java
and TaskExecutor.java and allow different developers to develop their own VMProvisioner and TaskExecutor. -- ongoing work
20) The default executor should be the local executor. --ongoing work
The current version is implemented based on JAVA8. Since the ClassLoader cannot be cast into URLClassLoader from JAVA 9.
1) Release day: to be discussed --- 07/01/19 2) Video: how to create access keys -- Done 3) Integrate with Ishtiaq's workflow executor alpha and workflow planner E2C2D (not SGX-E2C2D) --Done 4) Implement the Dropbox component so that DATAVIEW does not rely on another external script. (depends on a PHP that is outside of DATAVIEW). (ongoing work) 5) Use provenance to modify the configuration files for workflow scheduling algorithm LPOD. (ongoing work) 6) Deploy DATAVIEW as a web site. (Done) 7) We will use json format for config.json file, which is the file that configures the whole DATAVIEW system, the following properties of the system are considered:
a) WorkflowInputDir_DropboxFileSystem=DATAVIEW_INPUT b) WorkflowInputDir_LocalFileSystem=. c) accessKeyID and secretAccessKey, VMImageID,
d) confidentialTask = taskname e) <VMType, IP> The directory locations to put tasks, input data, and output data. -- ongoing work 8) Use Javadoc comments so that we can generate DATAVIEW APIs easily. ----- ongoing work 9) workflow structure visualization ---- Done 11) Provenance node miss spelling --Done 12) full name for EST ----- Done 13) In Everywhere token --> dropboxToken --Done 14) send the workflow final outputs to the corresponding files to the DATAVIEW master node. --Done 15) Introduce a data structure called taskExecutionStatus in JSON. -- Done 16) Implement the WorkflowExecutor_localBeta -- Done 17） Change to dataMove() method in taskExecutor.java --Done 18) Code refactoring: Where should these files belong? CmdLineDriver.java, MSGClient.java, MakeMachinesReady.java, MoveDataToCloud.java, MoveToCloud.java. -- ongoing work 19) Abstract classes for VMProvisioner and TaskExecutor: What common methods should we have for VMProvisioner.java and TaskExecutor.java? Maybe we can define an abstract VMProvisioner.java and TaskExecutor.java and allow different developers to develop their own VMProvisioner and TaskExecutor. -- ongoing work 20) The default executor should be the local executor. --ongoing work
The current version is implemented based on JAVA8. Since the ClassLoader cannot be cast into URLClassLoader from JAVA 9.