search

neurodata / ndpaper

NeuroData Paper Landing page
http://ndpaper.neurodata.io
Apache License 2.0
0 stars 3 forks source link

validate proof-of-concept bias assessment run #81

Closed wrgr closed 8 years ago

wrgr commented 8 years ago

Code is here: https://github.com/neurodata/ndpaper/tree/gh-pages/spatialsynapses Read: https://github.com/neurodata/ndpaper/blob/gh-pages/spatialsynapses/bias_validation.md Run: example script Data blocks (for fun) are here: https://drive.google.com/open?id=0B6iZJMt9RjNiRVYxVzlseE14bVU

jovo commented 8 years ago

do you have the estimated # of synapses for each of those 10 data blocks? i don't have a sense of how hard it is, so i am a bit lost on what a reasonable number of blocks is...

wrgr commented 8 years ago

I see. I don't - this is a qualitative assessment.

nominally, these are 5x5x5 micron blocks, u = 1.56, std = 0.32. And we are evaluating density estimates between these values, corresponding to this number of synapses (roughly, disregarding masking). So far with these bounds I've been able to distinguish between the two, but it's challenging.

mean - 2_stdev and mean - 1_stdev: 115-155 synapses mean + 1_stdev and mean 2_stdev: 235-275 synapses

We restrict to these bounds to minimize outliers created from masking errors or spurious counts.

jovo commented 8 years ago

oh sorry, i meant the estimated by your algorithm, not manually. i want to compare you qualitative judgement with the algorithm, rather than merely you saying that you got them all right. does that make more sense?

wrgr commented 8 years ago

I think my answer above answers this, right? We select blocks based on densities computed from VESICLE classification...

These densities can be straightforwardly mapped to an estimated number of synapses based on block size (disregarding masking).

I'm not sure what you are asking I guess.

wrgr commented 8 years ago

For now should I proceed truthing these 100 blocks? O(1 full day of work), so prefer not to do if it's premature, but eager to get this result knocked out.

jovo commented 8 years ago

i think so.

On Wed, Aug 17, 2016 at 6:22 PM, William Gray Roncal < notifications@github.com> wrote:

For now should I proceed truthing these 100 blocks? O(1 full day of work), so prefer not to do if it's premature, but eager to get this result knocked out.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndpaper/issues/81#issuecomment-240567629, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjcgAWLvjSOCnaagYpdcwmBW7fAUDdks5qg4m4gaJpZM4JmxjK .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

jovo commented 8 years ago

ps - the answer to my question is a 3 column 10 row csv file, with:

id # synapses estimated by alg whether willgray says is high or low
blah blah blah
blah blah blah
wrgr commented 8 years ago

i can get something that is pretty close - I recorded the synapse densities but not the counts. This can be approximately mapped to count by multiplying by volume size (disregards masking). So will do that for now!

(i don't cleanly have column 2, and at this point can't regenerate without starting over.)

jovo commented 8 years ago

ok, no clean col 2, i think no need for table. was mostly asking if i could have that.

jovo commented 8 years ago

is this "Inspect sample run and verify that an independent evaluator can compute accuracy?" just me looking at the movies and seeing how i feel? or doing something else (eg, running code)

wrgr commented 8 years ago

let's chat about it! I wanted you to look through a couple of the movies and see if you think the task is reasonable, and at a minimum run the sample code so that the current effort is useful.

Truthing has turned out to be very time intensive, so fingers crossed that this latest run of 100 samples is good!

jovo commented 8 years ago

ok, which sample code? there are lots of codes...

wrgr commented 8 years ago

see top of issue. :) there's a README with a line of code to run.

jovo commented 8 years ago

ah, i see, i failed to understand that the readme provided me instructions.

wrgr commented 8 years ago

oh - i see. this whole issue was yours, so i wasn't sure what we were waiting on. I already did mine to set all this up!

jovo commented 8 years ago

no worries, ffr, if there was a tick mark that said "run this code and tell me....", i would have known what you meant more. no worries, i will do it asap.

wrgr commented 8 years ago

it's much appreciated! #notstoppingscience

jovo commented 8 years ago

i don't know what to do. there is a command at the bottom of the readme file, presumably i need to pip install something? or download some code? can you provide more clear/explicit guidance?

wrgr commented 8 years ago

clone the repo. run the code, which is in the spatialsynapses folder. If this is still unclear, I can do better if you tell me how.

jovo commented 8 years ago 
102523260:~ jovo$ cd ~/Research/Projects/neurodata/ndpaper/
102523260:ndpaper jovo$ git pull
remote: Counting objects: 287, done.
remote: Total 287 (delta 31), reused 31 (delta 31), pack-reused 256
Receiving objects: 100% (287/287), 32.36 MiB | 5.60 MiB/s, done.
Resolving deltas: 100% (133/133), completed with 23 local objects.
From https://github.com/neurodata/ndpaper
   266be03..60d135e  gh-pages   -> origin/gh-pages
Your configuration specifies to merge with the ref 'master'
from the remote, but no such ref was fetched.
102523260:ndpaper jovo$ git branch
* master
102523260:ndpaper jovo$ python evaluate_bias.py plumbing_test_0817_v1.npz plumbing_test_0817_v1_will.csv
python: can't open file 'evaluate_bias.py': [Errno 2] No such file or directory
102523260:ndpaper jovo$ ls
README.md       neurocartography    spatialsynapses
analysis        paper           store
casestudy1      parse
explore         reproducibility
102523260:ndpaper jovo$ cd spatialsynapses/
102523260:spatialsynapses jovo$ python evaluate_bias.py plumbing_test_0817_v1.npz plumbing_test_0817_v1_will.csv
python: can't open file 'evaluate_bias.py': [Errno 2] No such file or directory
102523260:spatialsynapses jovo$
wrgr commented 8 years ago

can you ls for me? I see the file when I try those steps.

jovo commented 8 years ago

nothing was there. i was in master, not gh-pages. so, i switched and got this

102523260:ndpaper jovo$ git branch

* gh-pages

  master

102523260:ndpaper jovo$ ls

LICENSE explore.md spatialsyn_figs.ipynb

_includes index.html spatialsynapses

_layouts javascripts spatialsynapses.md

_source neurocartography store

analysis neurocartography.md store.md

analyze.md paper stylesheets

data.md parse.md web

102523260:ndpaper jovo$ cd spatialsynapses

102523260:spatialsynapses jovo$ ls

bias_exploration_v2.ipynb plumbing_test_0817_v1_will.csv

bias_validation.md put_mask.py

evaluate_bias.py put_mask23.py

figs put_mask23_brainviz.py

get_matrix.py spatialsyn_figs.ipynb

merged_mask_v5_layer23only_DSP.csv synapse_viz.ipynb

merged_mask_v6_layerall_DSP.csv test_bock_matrix.sh

plumbing_test_0817_v1.npz

102523260:spatialsynapses jovo$ python evaluate_bias.py
plumbing_test_0817_v1.npz plumbing_test_0817_v1_will.csv

***********************************************************************

Test Run: plumbing_test_0817_v1

Test Description: 8172016 10 samples, mean -1, -2, mean +1 + 2 std, 90%
unmasked

Accuracy: 100.0%

Total Samples: 10

True Positives: 5

True Negatives: 5

False Positives: 0

False Negatives: 0

***********************************************************************

102523260:spatialsynapses jovo$

On Mon, Aug 22, 2016 at 10:18 PM, William Gray Roncal < notifications@github.com> wrote:

can you ls for me? I see the file when I try those steps.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndpaper/issues/81#issuecomment-241608096, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjchL-lsMs0GKWuXI-rBZzCZmggVk0ks5qiliRgaJpZM4JmxjK .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

wrgr commented 8 years ago

Hey - that's sweet! Awesome sauce.

wrgr commented 8 years ago

I think we agreed to kill master branch (it doesn't exist in the github remote). gh-pages is default. If you prefer a different org, complain here or open an issue and I'll fix us!

jovo commented 8 years ago

i thought so too, but it is still there. no action necessary at this time.

On Mon, Aug 22, 2016 at 10:26 PM, William Gray Roncal < notifications@github.com> wrote:

I think we agreed to kill master branch (it doesn't exist in the github remote). gh-pages is default. If you prefer a different org, complain here or open an issue and I'll fix us!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndpaper/issues/81#issuecomment-241609103, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjcpDyTSXzkgqjtc7m4VM0UUerQ6b3ks5qilpFgaJpZM4JmxjK .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York