Import a popular neuro mailing list such as FreeSurfer into Neurostars

[brainstorm]

During the OHBM hackathon in Berlin, a few users expressed their interest in having a more interactive version of the FreeSurfer mailing list:

http://www.mail-archive.com/freesurfer@nmr.mgh.harvard.edu/info.html

That would allow users to:

1. Quickly search through archives.
2. Rank answers for relevance (collaborative filtering).
3. Filter by tag(s).

For that import to happen we should:

• [x] Ask the original archive owners if that's ok for them.
• [x] Identify the attributes we wish to migrate (usernames, thread structure...).
• [ ] Pre-filter relevant questions to have a good bootstrapping.
• [x] Setup an email mechanism similar to Gmane so that new emails get into neurostars automatically, not requiring periodic imports.

@satra, @nicholsn, feedback is welcome. I thought about starting with this:

http://stackoverflow.com/questions/22006616/how-convert-an-mbox-to-a-json-structure

While @nimiq is finishing up the deployment roles:

https://github.com/nimiq/ansible-neurostars.org

[ialbert]

Biostar already has the ability to parse mailing lists in the mbox format. Moreover one can even use emails to reply to posts (that is replying to emails will be stored as answers/comments).

The Galaxy Biostar support forum has been recently migrated from a mailing list, 8 years worth of post were migrated: https://biostar.usegalaxy.org/ and the Bioconductor team is also investigating migrating their mailinglist over to Biostar: https://support.bioconductor.org/

the command that does this is called import_mbox see help for it::

python manage.py import_mbox -h 

also there is a tagging mechanism added as well that tags posts that match a regular expression with a certain tag (this too was applied to the Galaxy import)

The rules of migration are the following. Original email is a post, all replies to the original email are answers, replies to replies become comments.

[brainstorm]

Oh, nice, thanks @ialbert, I should really take a second round on biostars codebase reading, I shall not miss good stuff like this.

@nimiq, hopefully we will have the roles/deployment scripts ready by next week, if not, we should go ahead and deploy manually (again, I hope we don't have to). After importing the old posts I will definitely would like to give this a shot myself, just so you know.

[puntonim]

I've done a lot of work with the deployment. Yesterday night I completed it. Today I will fix the last details and publish it. It is cool thing, but I guess it was a bad idea starting the GSoC with this topic, because it took so long.

[brainstorm]

Indeed, thanks @nimiq for the good work and discussions! :+1:

[brainstorm]

@ialbert I have been reading about Amazon SES, but apparently they only offer an outbound smtp service, I guess you are running another smtp server for inbound requests for the "reply to posts" feature? Do you use your institutional smtp server or a generic Gmail account?

In order to break away from Amazon services, we would prefer to keep the email backend as plain old smtp, did you run into problems trying to go this way?

[ialbert]

The default setup assumes that a local SMTP server has been set up. We use SES for convenience mostly and to ensure better email delivery.

I have added more details on sending and receving emails to the docs:

http://docs.biostars.org/en/latest/install.html#sending-emails

On Wed, Jun 18, 2014 at 10:18 AM, Roman Valls Guimerà < notifications@github.com> wrote:

@ialbert https://github.com/ialbert I have been reading about Amazon SES, but apparently they only offer an outbound smtp service, I guess you are running another smtp server for inbound requests for the "reply to posts" feature? Do you use your institutional smtp server or a generic Gmail account?

In order to break away from Amazon services, we would prefer to keep the email backend as plain old smtp, did you run into problems trying to go this way?

— Reply to this email directly or view it on GitHub https://github.com/INCF/biostar-central/issues/36#issuecomment-46441131.

[brainstorm]

@satra, @nicholsn, @yarikoptic, @binarybottle. After getting the email notifications fixed for neurostars2, I am going to import the FSL mailing list archives (142MiB mbox file, ~38.000 messages) into http://neurostars.org... Now, are we sure we want to do this? I'm of the opinion that a bit of filtering beforehand would help, but I'm not sure about the best way to do that.

Here's the script I used to fetch them:

https://gist.github.com/brainstorm/de2904e0807c0f436d0b

I'll be using the import script pointed by Istvan:

https://github.com/ialbert/biostar-central/blob/master/biostar/server/management/commands/import_mbox.py

[satra]

we should check licenses if there are any first.

[nicholsn]

also, is there a staging area where we can get a feel for what this looks like when it it imported?

On Thu, Jun 19, 2014 at 6:45 AM, Satrajit Ghosh notifications@github.com wrote:

we should check licenses if there are any first.

— Reply to this email directly or view it on GitHub https://github.com/INCF/biostar-central/issues/36#issuecomment-46561687.

[brainstorm]

@satra True, I sent the following a few days ago to freesurfer-owner@nmr.mgh.harvard.edu to get some type of consent, I could not find a clear statement on their wiki nor list, let me know if you do:

Hello FSL owner,

My name is Roman, I work at the International Neuroinformatics Coordinating Facility.

The reason for emailing you is that I’ve enacted a stackoverflow-like website for neuroscience:

http://neurostars.org/

And I’ve recently received some requests from FSL users to have some (if not all) of the archives imported to neurostars:

https://github.com/INCF/biostar-central/issues/36

I would like to know if that is ok for the FSL community before doing so via pipermail archives:

https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/

Best,
Roman

@nicholsn Yes, http://staging.neurostars.org, I'll share the credentials with you guys, for now it just looks like the live site though.

[binarybottle]

Great idea!

cheers, @rno

On Thu, Jun 19, 2014 at 9:42 AM, Roman Valls Guimerà < notifications@github.com> wrote:

@satra https://github.com/satra, @nicholsn https://github.com/nicholsn, @yarikoptic https://github.com/yarikoptic, @binarybottle https://github.com/binarybottle. After getting the email notifications fixed for neurostars2, I am going to import the FSL mailing list archives (142MiB mbox file, ~38.000 messages) into http://neurostars.org... Now, are we sure we want to do this? I'm of the opinion that a bit of filtering beforehand would help, but I'm not sure about the best way to do that.

Here's the script I used to fetch them:

https://gist.github.com/brainstorm/de2904e0807c0f436d0b

I'll be using the import script pointed by Istvan:

https://github.com/ialbert/biostar-central/blob/master/biostar/server/management/commands/import_mbox.py

— Reply to this email directly or view it on GitHub https://github.com/INCF/biostar-central/issues/36#issuecomment-46561301.

[rno]

aaaaaannnnd again :smile:

[yarikoptic]

aggregation of disjoint neurosoftware communities discussions under the same roof sounds REALLY cool idea, BUT what would be the goal???
neurostars will not provide a bidirectional gateway, right? fetching archives is cool, but as a result, if anyone would like to continue discussion on the neurostars instead of original forum, without a flow of replies back to the list it might just split those communities/interactions and thus annoy those projects' teams? or am I wrong? P.S. great call @satra on licensing considerations

[nicholsn]

I think the goal would be to capture all that knowledge that is lost into the mailing list ether and turn it into something that is searchable and useful for finding previously answered questions, right?.

That being said, we'd almost need to automatically tag the posts in some way to make it more digestible, which is why I'd be curious to see what the uploaded data looks like in a staging area.

Also, i can see where it might be annoying that the replies don't flow back to the mailing list, but isn't that less work for those answering the questions? Isn't this meant to be a demonstration of why mailing lists should move to more stack overflow style of Q&A?

On Thu, Jun 19, 2014 at 7:41 AM, Yaroslav Halchenko < notifications@github.com> wrote:

aggregation of disjoint neurosoftware communities discussions under the same roof sounds REALLY cool idea, BUT what would be the goal???

neurostars will not provide a bidirectional gateway, right? fetching archives is cool, but as a result, if anyone would like to continue discussion on the neurostars instead of original forum, without a flow of replies back to the list it might just split those communities/interactions and thus annoy those projects' teams? or am I wrong? P.S. great call @satra https://github.com/satra on licensing considerations

— Reply to this email directly or view it on GitHub https://github.com/INCF/biostar-central/issues/36#issuecomment-46568775.

[puntonim]

We might even think of automatically subscribe (or ask for it) all people involved in the mailing list to the new Q&A thread in Neurostars. This way everyone gets notified via email when a new answer or comment is posted on Neurostars. Or is this too "aggressive"?

[satra]

this is really intended to create a cultural shift in the biomedical imaging community. i still see many posts on fsl and spm lists which are identical either in content or in concept. moving over to a stackoverflow like forum to me makes a lot more sense. in the future, we have also imagined that biostars and neurostars will link to each other because information is not really segregated always. it made sense when domains were more silo-ed, but i think we as a community and especially as INCF can take a step towards coordinating information.

i understand that things will take some time, but if google searches repeatedly get people to useful content, people will start migrating.

now given that the number of people responding to emails on freesurfer, fsl and spm mailing lists are typically less than 5 per list, we might even get those people to eventually migrate. but i really don't want to think of neurostars as a brain imaging knowledge base. it should also start attracting all other communities of neuroscience.

[nicholsn]

i.e., world domination =)

On Thu, Jun 19, 2014 at 9:57 AM, Satrajit Ghosh notifications@github.com wrote:

this is really intended to create a cultural shift in the biomedical imaging community. i still see many posts on fsl and spm lists which are identical either in content or in concept. moving over to a stackoverflow like forum to me makes a lot more sense. in the future, we have also imagined that biostars and neurostars will link to each other because information is not really segregated always. it made sense when domains were more silo-ed, but i think we as a community and especially as INCF can take a step towards coordinating information.

i understand that things will take some time, but if google searches repeatedly get people to useful content, people will start migrating.

now given that the number of people responding to emails on freesurfer, fsl and spm mailing lists are typically less than 5 per list, we might even get those people to eventually migrate. but i really don't want to think of neurostars as a brain imaging knowledge base. it should also start attracting all other communities of neuroscience.

— Reply to this email directly or view it on GitHub https://github.com/INCF/biostar-central/issues/36#issuecomment-46587282.

[brainstorm]

Heh, funnily enough, there was such a milestone defined prior to @satra's comment :P

https://github.com/INCF/biostar-central/issues?milestone=3&state=open

Now, I haven't had feedback from FSL since almost 2 weeks... @satra, since you've been a prolific FSL user since 2004 according to the list archives, could you please pick a brain or two and possibly get a :+1: from FSL owners?

[satra]

@brainstorm : did you mean freesurfer or fsl or both?

[brainstorm]

Sorry, freesurfer :-S

But while you are at it, FSL might be a good candidate too :)

[brainstorm]

This issue was moved to brainstorm/biostar-central#1