Refactoring for CI

[Cerfoglg]

Warning: massive amount of changes done.

works on #5

• [ ]  General Comment
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> - Travis only on dev branch
• Put back whitelist in travis
• Take benchflow.cql from repo
• Dockerfile to Dockerfile.c
• Eventually using our Spark image
• minio port and keyspace not in conf
• transformation empty
• urllib calls in its own function
• if else in properties
• config file not SUT dependant
• use correct alluxio filename structure (cassandra primary key)
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> @Cerfoglg still to do:
• Dockerfile.c --> Dockerfile.ci
• Use our Spark image
• transformation empty
• config file not SUT dependant
• use correct alluxio filename structure (cassandra primary key)
• Everything mentioned in https://github.com/benchflow/analysers/pull/57, that also apply here Then we first need to integrate https://github.com/benchflow/data-transformers/pull/50 because the repository structure seems different (e.g., we don't have the conf folder anymore). Once done ping me, so that I can review the PR.
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Should be ready now
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Now it should actually
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> We currently skip to fix Codacy and CodeClimate errors.
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Method could be a function
• [ ]  File: test/spark/cassandraTest.py:L1-54
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import gzip
• [ ]  File: test/spark/cassandraTest.py:L1-54
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import sys
• [ ]  File: test/spark/cassandraTest.py:L1-54
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused RowFormat imported from pyspark_cassandra
• [ ]  File: test/spark/cassandraTest.py:L1-54
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import io
• [ ]  File: test/spark/cassandraTest.py:L1-54
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import uuid
• [ ]  File: test/spark/cassandraTest.py:L1-46
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import json
• [ ]  File: data-transformers/transformers/statsTransformer.py:L95-164
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused variable 'SUTName'
• [ ]  File: data-transformers/transformers/statsTransformer.py:L95-164
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused variable 'table'
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Method could be a function
• [ ]  File: data-transformers/transformers/statsTransformer.py:L95-165
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: No exception type(s) specified
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Method could be a function
• [ ]  File: data-transformers/transformers/mysqlTransformer.py:L1-30
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused variable 'transformations'
• [ ]  File: data-transformers/transformers/mysqlTransformer.py:L1-30
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused variable 'transformations'
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Method could be a function
• [ ]  File: data-transformers/transformers/mysqlTransformer.py:L1-30
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused variable 'transformations'
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Method could be a function
• [ ]  File: data-transformers/transformers/statsTransformer.py:L1-16
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: Unused import yaml
• [ ]  File: data-transformers/transformers/statsTransformer.py:L97-168
• <img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Issue found: No exception type(s) specified
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> It's not always clear what exception happens here, so leaving it unspecified for now.
• [ ]  File: data-transformers/transformations/dataTransformations.py:L1-4
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> I'm using the functions in Transformations by doing an eval to call the right one based on the configuration file for the transformation to apply.
• [ ]  File: data-transformers/transformers/statsTransformer.py:L1-16
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> Will be used, so might as well keep it in for now.
• [ ]  File: data-transformers/transformers/mysqlTransformer.py:L1-30
• <img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> I actually do use it, I do an eval()

  

[Cerfoglg]

• Travis only on dev branch
• Put back whitelist in travis
• Take benchflow.cql from repo
• Dockerfile to Dockerfile.c
• Eventually using our Spark image
• minio port and keyspace not in conf
• transformation empty
• urllib calls in its own function
• if else in properties
• config file not SUT dependant
• use correct alluxio filename structure (cassandra primary key)

[VincenzoFerme]

@Cerfoglg still to do:

• Dockerfile.c --> Dockerfile.ci
• Use our Spark image
• transformation empty
• config file not SUT dependant
• use correct alluxio filename structure (cassandra primary key)
• Everything mentioned in https://github.com/benchflow/analysers/pull/57, that also apply here

Then we first need to integrate https://github.com/benchflow/data-transformers/pull/50 because the repository structure seems different (e.g., we don't have the conf folder anymore).

Once done ping me, so that I can review the PR.

[Cerfoglg]

@VincenzoFerme Should be ready now

[Cerfoglg]

@VincenzoFerme Now it should actually

[VincenzoFerme]

We currently skip to fix Codacy and CodeClimate errors.