search

neurodata / ndgrutedb

Pipeline and tools for estimating human connectomes from Diffusion, Structural, and Functional MRI
http://neurodata.io
Apache License 2.0
29 stars 13 forks source link

Cleanup graphs with 0 edges #219

Closed disa-mhembere closed 7 years ago

disa-mhembere commented 7 years ago

Delete or move from brainstore then run let me know so I can update the website

The see 0 edge graphs => http://openconnecto.me/graph-services/download/?sort=numedge

I think a script to check if degree == 0 should be fast if you only read the headers. You could build off this script to read graphml headers

disa-mhembere commented 7 years ago

@gkiar I've received 3 emails in the last two weeks regarding these graphs so I want to know which of these are wrong. I modified the ingest to cleanup 0 degree and missing graphs, but there are graphs with 1 and 7 edges which still seems wrong. Can you comment on this?

gkiar commented 7 years ago

Which graphs/how do I find them? I'm generally not pushing any human graphs back to your service so they are all out of date, anyways. Who is finding them there? I'm moving towards hosting them in S3 and having our own static data table which points people to the data.

On Feb 8, 2017 23:02, "Disa Mhembere" notifications@github.com wrote:

@gkiar https://github.com/gkiar I've received 3 emails in the last two weeks regarding these graphs so I want to know which of these are wrong. I modified the ingest to cleanup 0 degree and missing graphs, but there are graphs with 1 and 7 edges which still seems wrong. Can you comment on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndgrutedb/issues/219#issuecomment-278541221, or mute the thread https://github.com/notifications/unsubscribe-auth/AEqDWKEAluaqD8Idfql-zEaxmribOJi8ks5rao-2gaJpZM4J7GQ2 .

disa-mhembere commented 7 years ago

Which graphs/how do I find them? The link i posted before :) here it is again. Note that many of the 0 deg ones will disappear within a couple of days since the cleanup script is running only on 16 out of the cores.

People still find, pull & use these graphs and services judging from the the logs and google analytics so maybe we can just delete the old graphs or replace them with the newer ones?

gkiar commented 7 years ago

This is what I see at that link: screen shot 2017-02-10 at 11 04 46 am

I find the current interface for the graph table to not make sense for human graphs anymore. There are too many of them for a table with only 20 entries per page to effectively work through, so I suggest we remove human MR graphs from the service and provide a link in its place saying they've moved to []().

The reason I don't want to replace the old graphs is because then provenance is lost. I'd rather have all graphs available, but this service doesn't currently allow for that and I don't want you to have to sink time into making that the case.

disa-mhembere commented 7 years ago

Think you have the wrong URL :) http://www.openconnecto.me/graph-services/download/?sort=numedge.

You can also just click and order by number of edges to see 1 degree graphs. I would also verify these with the newest version of your graphgen that this isn't true. I would think it's not right ?

You can give me the URL where you host them. I'll delete the old ones :)

On Feb 10, 2017 11:09 AM, "Greg Kiar" notifications@github.com wrote:

This is what I see at that link: [image: screen shot 2017-02-10 at 11 04 46 am] https://cloud.githubusercontent.com/assets/4883288/22833663/c11f5bde-ef80-11e6-8a7f-0d464f522ba9.png

I find the current interface for the graph table to not make sense for human graphs anymore. There are too many of them for a table with only 20 entries per page to effectively work through, so I suggest we remove human MR graphs from the service and provide a link in its place saying they've moved to .

The reason I don't want to replace the old graphs is because then provenance is lost. I'd rather have all graphs available, but this service doesn't currently allow for that and I don't want you to have to sink time into making that the case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndgrutedb/issues/219#issuecomment-278985247, or mute the thread https://github.com/notifications/unsubscribe-auth/ABtfJN7rXkSfpIw47vs55L0y4vMdxkCzks5rbItjgaJpZM4J7GQ2 .

gkiar commented 7 years ago

I just clicked your previous link, thanks for fixing it :)

We know that in previous versions of the pipeline some of the graphs had very low degree/edge counts, and these have been corrected in newer versions. I will give you a link to the new graph hosting location once I populate it more fully - it's on my stack for next week. :)

disa-mhembere commented 7 years ago

Yeah I must have just sent you the wrong link or something I didn't fix anything lol.

OK just put the link in this message so I can see it and we can close this issue.

On Feb 10, 2017 11:24 AM, "Greg Kiar" notifications@github.com wrote:

I just clicked your previous link, thanks for fixing it :)

We know that in previous versions of the pipeline some of the graphs had very low degree/edge counts, and these have been corrected in newer versions. I will give you a link to the new graph hosting location once I populate it more fully - it's on my stack for next week. :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndgrutedb/issues/219#issuecomment-278989637, or mute the thread https://github.com/notifications/unsubscribe-auth/ABtfJNWB1VAawc6EFtIVMcarJDQg1fG3ks5rbI8XgaJpZM4J7GQ2 .

gkiar commented 7 years ago

New graph hosting/documentation location: http://docs.neurodata.io/mri.neurodata.io/

disa-mhembere commented 7 years ago

I had already cleaned these and should have closed it