make_ML_HDF5 - Githubissues

hermands commented 9 years ago

Subcommand:

Adopt schema of run_ML_on_TubeCases.py, but change so that takes input of clinical annotations, HDF5_FEATURE_FILE, and db
Pick cases and case_tube_idx's to keep based on query and intersection of various tables
Store excluded cases, errors
Push all information to ML_DATA_HDF5_IO

Class: ML_DATA_HDF5_IO

Inputs: -- feature df -- annotation df -- errors/exclusions df -- features/bin/methods + version
Methods: -- I/O all

davidpng commented 9 years ago

I just ran ./flowanal.py -vvv make_ML_HDF5 --tubes "Hodgkin"

Running into a problem with db.query? Any suggestions? Error message below: INFO query_database 39 Querying... Traceback (most recent call last): File "./flowanal.py", line 7, in sys.exit(main(sys.argv[1:])) File "/home/ngdavid/Repository/FCS_Database/FlowAnal/scripts/main.py", line 117, in main return action(arguments) File "/home/ngdavid/Repository/FCS_Database/FlowAnal/subcommands/make_ML_HDF5.py", line 59, in action annotation_df = db.query(exporttype='df', getCaseAnnotations=True, _vars(args)).results File "/home/ngdavid/Repository/FCS_Database/FlowAnal/database/FCS_database.py", line 94, in query return queryDB(self, exporttype=exporttype, _kwargs) File "/home/ngdavid/Repository/FCS_Database/FlowAnal/database/query_database.py", line 66, in init '='.join(key, value) for key, value in kwargs.items()) ValueError: <generator object at 0x42fb640>

hermands commented 9 years ago

I'll take a peak tomorrow

hermands commented 9 years ago

I am missing the point. What is the goal of this script?

How is it different from the make_ML_input I made on Friday/Saturday?

hermands commented 9 years ago

Issue with query is probably that getCaseAnnotations is not a listed function. This is a remnant of our previous model in which the case annotations came from the db. I have not written the function that exports annotation from db. make_ML_input is just built to take the text files.

use 1) dev/test_run.sh 2) direct call to make features. 3) dev/test_run_custom.sh

davidpng commented 9 years ago

Sorry, I didn't notice make_ML_input and make_ML_HDF5 were similar files. Your comment on Issue 46 suggested this was not yet done (as it was assigned to me) so I probably duplicated this work.

On Thu, Jan 15, 2015 at 7:43 AM, Daniel Herman notifications@github.com wrote:

Issue with query is probably that getCaseAnnotations is not a listed function. This is a remnant of our previous model in which the case annotations came from the db. I have not written the function that exports annotation from db. make_ML_input is just built to take the text files.

use 1) dev/test_run.sh 2) direct call to make features. 3) dev/test_run_custom.sh

— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/46#issuecomment-70104943 .

hermands commented 9 years ago

I had assigned to you because I was done and handing it off to you to work with / try out. I should have commented as such. Sorry.

-Dan

On Jan 15, 2015, at 7:54 AM, David Ng notifications@github.com wrote:

Sorry, I didn't notice make_ML_input and make_ML_HDF5 were similar files. Your comment on Issue 46 suggested this was not yet done (as it was assigned to me) so I probably duplicated this work.

On Thu, Jan 15, 2015 at 7:43 AM, Daniel Herman notifications@github.com wrote:

Issue with query is probably that getCaseAnnotations is not a listed function. This is a remnant of our previous model in which the case annotations came from the db. I have not written the function that exports annotation from db. make_ML_input is just built to take the text files.

use 1) dev/test_run.sh 2) direct call to make features. 3) dev/test_run_custom.sh

— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/46#issuecomment-70104943 .

— Reply to this email directly or view it on GitHub.

hermands commented 9 years ago

I should have been more careful with names. I spent a lot of time to get it to work...sorry if you did also. We walked through this script together on Friday.

-Dan

On Jan 15, 2015, at 7:54 AM, David Ng notifications@github.com wrote:

Sorry, I didn't notice make_ML_input and make_ML_HDF5 were similar files. Your comment on Issue 46 suggested this was not yet done (as it was assigned to me) so I probably duplicated this work.

On Thu, Jan 15, 2015 at 7:43 AM, Daniel Herman notifications@github.com wrote:

Issue with query is probably that getCaseAnnotations is not a listed function. This is a remnant of our previous model in which the case annotations came from the db. I have not written the function that exports annotation from db. make_ML_input is just built to take the text files.

use 1) dev/test_run.sh 2) direct call to make features. 3) dev/test_run_custom.sh

— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/46#issuecomment-70104943 .

— Reply to this email directly or view it on GitHub.

davidpng commented 9 years ago

No, its my fault, I should have remembered we walked through it. Its actually a good thing I went through it because I had to modify some of the HDF5_IO functions to handle panda series objects and incorporate the passage of bin descriptions and other meta data from Feature_IO to MLInput_IO hdf5 files.

On Thu, Jan 15, 2015 at 8:08 AM, Daniel Herman notifications@github.com wrote:

I should have been more careful with names. I spent a lot of time to get it to work...sorry if you did also. We walked through this script together on Friday.

-Dan

On Jan 15, 2015, at 7:54 AM, David Ng notifications@github.com wrote:

Sorry, I didn't notice make_ML_input and make_ML_HDF5 were similar files. Your comment on Issue 46 suggested this was not yet done (as it was assigned to me) so I probably duplicated this work.

On Thu, Jan 15, 2015 at 7:43 AM, Daniel Herman notifications@github.com

wrote:

Issue with query is probably that getCaseAnnotations is not a listed function. This is a remnant of our previous model in which the case annotations came from the db. I have not written the function that exports annotation from db. make_ML_input is just built to take the text files.

use 1) dev/test_run.sh 2) direct call to make features. 3) dev/test_run_custom.sh

— Reply to this email directly or view it on GitHub < https://github.com/davidpng/FCS_Database/issues/46#issuecomment-70104943> .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/davidpng/FCS_Database/issues/46#issuecomment-70109596 .

hermands commented 9 years ago

Needs some QC/QI.

Meta data is not being added.
Overall success needs to be evaluated

davidpng / FCS_Database

make_ML_HDF5 #46