Publish Graphs as a Research Note

[IshaGupta18]

Is your feature request related to a problem? Please describe. Once we complete the integration with plots2, one of the features for Public Lab users could be to publish the created graphs as a Research Note.

Propose ideas to implement this feature This is how I thought of doing it while I was writing my proposal.

However, I am not sure how to go about it exactly.

Do we store the graphs as images and render them in notes or how would we implement the partial thing exactly.

Additional context

I would love it if anyone could suggest a flow of things here. On how should we go about it.

@jywarren @namangupta01 @rexagod @gauravano @sagarpreet-chadha @IgorWilbert

[jywarren]

Hi! Notice how when you drag a CSV into https://stable.publiclab.org/post, you get this power tag:

That's the "rich text" view. In Markdown, it's simply: [graph:/i/21643.csv]

When you hit Publish, you see this:

https://stable.publiclab.org/notes/warren/07-03-2019/test-csv

That's because this power tag, in format [graph:/i/21643.csv], gets converted into the following HTML/JS code:

<canvas class="inline-graph" id="graph-grid-101"></canvas>
<p class="graph-utils"><small><a href="/i/21643.csv"><i class="fa fa-download"></i> CSV</a></small></p>
<style>
.graph-utils {
text-align: right;
}
.graph-utils a,
.graph-utils i {
color: #aaa;
}
</style>
<script>
(function(){
graphUrl("/i/21643.csv", "graph-grid-101");
})()
</script>
</p>

The substitution is done here: https://github.com/publiclab/plots2/blob/50f97ea138acaeb4db641f50be1835886945d2ef/app/models/concerns/node_shared.rb#L38-L54

And the output is driven from this template (as referenced in the substitution method above, which scans through the note body):

https://github.com/publiclab/plots2/blob/50f97ea138acaeb4db641f50be1835886945d2ef/app/views/grids/_graph.html.erb#L1-L16

See how this fits together?

[jywarren]

Similarly, you could create a new graph powertag, like [simple-graph:/i/example.csv] which would get inserted in the same way. Don't worry about the drag-and-drop part yet, I can show you later. (see here for more)

For now, just focus on how the powertag [graph:/i/example.csv] gets converted into a displayed graph. If you make your own template at /app/views/grids/_simple_graph.html.erb you can start setting this up!

We actually have a test demonstrating the existing graph integration here:

https://github.com/publiclab/plots2/blob/50f97ea138acaeb4db641f50be1835886945d2ef/test/unit/node_shared_test.rb#L170-L174

You could make a similar test to show your own powertag!

[IshaGupta18]

Wow this is some pretty great stuff here. I can see how this ties up together, but I think I might want to do it a little differently. Since the graphs are already made, we could create a Research Note similar to this one but with the already plotted graphs, instead of drag and drop (since it already exists). Let me study this a little more deeply and I think we'll be good to go with a similar thing then. I'll post more doubts once I begin with this! Thanks a ton!

[IgorWilbert]

@IshaGupta18 not sure if I understand how you want to do this. If there is no drag and drop to create a Research Note with a given graph, do you plan to attach the graph to the note with an "upload button", for example? If yes, why don't we do both? If I am not mistaken, in Image Sequencer it is possible to upload an image either selecting the file or using drag and drop. Maybe we could do the same but with graphs and "loading a preview" of them too. Also, if you need some help with UI or coding problems just ping ;)

[jywarren]

I think we could start by implementing the [simple-graph:____.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:____.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

[IshaGupta18]

That sounds pretty good, although a little tough, since we need to run the entire thing for the selection of columns and stuff in the drag drop box itself. What I suggest is, we create a "Publish as research note" button where the graphs are plotted, and from there itself it can pick up the plotted graphs and proceed in a similar manner as above. What do you think?

On Mon, Jul 8, 2019, 9:34 PM Jeffrey Warren notifications@github.com wrote:

I think we could start by implementing the [simple-graph:.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZZQHYSGPQ2GDIG44HTP6NQRTA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNR7WY#issuecomment-509288411, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXHQZYGNLBLGTSDH7PQ2MTP6NQRTANCNFSM4H5H5UNA .

[jywarren]

Oh, i see. So the configuration of SDG would need to be added to the short-code perhaps? How would that work? What are the parameters that would need to be stored based on the selection of columns and such?

On Mon, Jul 8, 2019 at 12:14 PM Isha Gupta notifications@github.com wrote:

That sounds pretty good, although a little tough, since we need to run the entire thing for the selection of columns and stuff in the drag drop box itself. What I suggest is, we create a "Publish as research note" button where the graphs are plotted, and from there itself it can pick up the plotted graphs and proceed in a similar manner as above. What do you think?

On Mon, Jul 8, 2019, 9:34 PM Jeffrey Warren notifications@github.com wrote:

I think we could start by implementing the [simple-graph:.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZZQHYSGPQ2GDIG44HTP6NQRTA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNR7WY#issuecomment-509288411 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AJXHQZYGNLBLGTSDH7PQ2MTP6NQRTANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6JZ2DMB4XJ6FHBIXBWDP6NRUXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNS25Y#issuecomment-509291895, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J2ILZWSFCJVP6XGVTDP6NRUXANCNFSM4H5H5UNA .

[IshaGupta18]

Yeah, we will create or use the current function for plotting the graph, which already has the required selections, so I believe that should work?

On Mon, Jul 8, 2019, 9:50 PM Jeffrey Warren notifications@github.com wrote:

Oh, i see. So the configuration of SDG would need to be added to the short-code perhaps? How would that work? What are the parameters that would need to be stored based on the selection of columns and such?

On Mon, Jul 8, 2019 at 12:14 PM Isha Gupta notifications@github.com wrote:

That sounds pretty good, although a little tough, since we need to run the entire thing for the selection of columns and stuff in the drag drop box itself. What I suggest is, we create a "Publish as research note" button where the graphs are plotted, and from there itself it can pick up the plotted graphs and proceed in a similar manner as above. What do you think?

On Mon, Jul 8, 2019, 9:34 PM Jeffrey Warren notifications@github.com wrote:

I think we could start by implementing the [simple-graph:.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZZQHYSGPQ2GDIG44HTP6NQRTA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNR7WY#issuecomment-509288411

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AJXHQZYGNLBLGTSDH7PQ2MTP6NQRTANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6JZ2DMB4XJ6FHBIXBWDP6NRUXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNS25Y#issuecomment-509291895 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AAAF6J2ILZWSFCJVP6XGVTDP6NRUXANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ6LJZHOYZEZV7PO4L3P6NSLXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNTL2Y#issuecomment-509294059, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXHQZZDRFLURK3KQ4XD5ULP6NSLXANCNFSM4H5H5UNA .

[jywarren]

well, can you list out the kinds of selections that need to be made, so we can think about what associated data would have to go with a .csv file? Thanks!

On Mon, Jul 8, 2019 at 1:07 PM Isha Gupta notifications@github.com wrote:

Yeah, we will create or use the current function for plotting the graph, which already has the required selections, so I believe that should work?

On Mon, Jul 8, 2019, 9:50 PM Jeffrey Warren notifications@github.com wrote:

Oh, i see. So the configuration of SDG would need to be added to the short-code perhaps? How would that work? What are the parameters that would need to be stored based on the selection of columns and such?

On Mon, Jul 8, 2019 at 12:14 PM Isha Gupta notifications@github.com wrote:

That sounds pretty good, although a little tough, since we need to run the entire thing for the selection of columns and stuff in the drag drop box itself. What I suggest is, we create a "Publish as research note" button where the graphs are plotted, and from there itself it can pick up the plotted graphs and proceed in a similar manner as above. What do you think?

On Mon, Jul 8, 2019, 9:34 PM Jeffrey Warren notifications@github.com wrote:

I think we could start by implementing the [simple-graph:.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZZQHYSGPQ2GDIG44HTP6NQRTA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNR7WY#issuecomment-509288411

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AJXHQZYGNLBLGTSDH7PQ2MTP6NQRTANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6JZ2DMB4XJ6FHBIXBWDP6NRUXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNS25Y#issuecomment-509291895

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AAAF6J2ILZWSFCJVP6XGVTDP6NRUXANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ6LJZHOYZEZV7PO4L3P6NSLXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNTL2Y#issuecomment-509294059 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AJXHQZZDRFLURK3KQ4XD5ULP6NSLXANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6J6N3ULJT6AG45T4HJLP6NX3LA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNXPVQ#issuecomment-509310934, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J3XVG2OMBEESJT437DP6NX3LANCNFSM4H5H5UNA .

[IshaGupta18]

Yes, just let me study about the short-code a little more, then I'll get back to you with the parameters. If I am not mistaken, we would be just needing the plotgraph function, which already has the selected parameters, that should do the trick. Thanks a lot!

On Tue, Jul 9, 2019, 12:21 AM Jeffrey Warren notifications@github.com wrote:

well, can you list out the kinds of selections that need to be made, so we can think about what associated data would have to go with a .csv file? Thanks!

On Mon, Jul 8, 2019 at 1:07 PM Isha Gupta notifications@github.com wrote:

Yeah, we will create or use the current function for plotting the graph, which already has the required selections, so I believe that should work?

On Mon, Jul 8, 2019, 9:50 PM Jeffrey Warren notifications@github.com wrote:

Oh, i see. So the configuration of SDG would need to be added to the short-code perhaps? How would that work? What are the parameters that would need to be stored based on the selection of columns and such?

On Mon, Jul 8, 2019 at 12:14 PM Isha Gupta notifications@github.com wrote:

That sounds pretty good, although a little tough, since we need to run the entire thing for the selection of columns and stuff in the drag drop box itself. What I suggest is, we create a "Publish as research note" button where the graphs are plotted, and from there itself it can pick up the plotted graphs and proceed in a similar manner as above. What do you think?

On Mon, Jul 8, 2019, 9:34 PM Jeffrey Warren < notifications@github.com> wrote:

I think we could start by implementing the [simple-graph:.csv] filtering in wikis/notes, without also doing drag/drop at the same time, as a first step. We could for example, as a first step, make it so you have to just drag/drop a CSV as you do now, but then change [graph:.csv] to [simple-graph:____.csv] and then your code would start working. Then later we could decide how we want to handle the dragging and dropping! How's that sound?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZZQHYSGPQ2GDIG44HTP6NQRTA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNR7WY#issuecomment-509288411

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AJXHQZYGNLBLGTSDH7PQ2MTP6NQRTANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6JZ2DMB4XJ6FHBIXBWDP6NRUXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNS25Y#issuecomment-509291895

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AAAF6J2ILZWSFCJVP6XGVTDP6NRUXANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ6LJZHOYZEZV7PO4L3P6NSLXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNTL2Y#issuecomment-509294059

, or mute the thread <

https://github.com/notifications/unsubscribe-auth/AJXHQZZDRFLURK3KQ4XD5ULP6NSLXANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AAAF6J6N3ULJT6AG45T4HJLP6NX3LA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNXPVQ#issuecomment-509310934 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AAAF6J3XVG2OMBEESJT437DP6NX3LANCNFSM4H5H5UNA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ7ZFZ26ICKEQHXLW4DP6OECXA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOANOY#issuecomment-509347515, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXHQZ5EU62OOJ62EBGT2C3P6OECXANCNFSM4H5H5UNA .

[IshaGupta18]

Okay, so I have been sleuthing through all the codes and files that come together and create the inline graph feature and I found a lot of interesting stuff, much of which I understood, but I still have some doubts one of them being, from where exactly are we sending this post request to /images

And in my study I found the entire trace: we go from show.html.erb to application_helper to node_shared which calls the partial _graph.html.erb which calls a functions in graph.js and presto! we get our inline graph. But what I am not getting is this post request. Is it coming from drag_drop.js or where? Because I see this

when I drag and drop a CSV, so where is this coming from exactly. I would be following a similar track to achieve this new feature, so thats why I need so much clarity. Please let me know the last bits and pieces of this puzzle!

@jywarren @gauravano @Souravirus @namangupta01

[IshaGupta18]

There are some links between this flow that I have written which are missing. For example where exactly is this tag created with a particular name like this and this format of values?

[jywarren]

Hi! The post request would come from dragdrop.js if from a comment textarea. But if from the editor at /post or /questions/new, then it's from the PublicLab.Editor project. Both post to the images controller to upload any filetype allowed. Then they get back a URL once it's uploaded. Hope that helps!

[jywarren]

And, you can manually create the tag. But, if you drag in, PublicLab.editor will autogenerate it for you.

https://github.com/publiclab/PublicLab.Editor/pull/137

And

https://github.com/publiclab/PublicLab.Editor/pull/141

[IshaGupta18]

@jywarren thanks a lot for the comment! However, I wanted to tell you that I am struggling a lot with this ie I am not able to clearly find a way to implement it, despite going through almost everything. Is there a way that I could to you or someone who has worked with editor and nodes, to achieve this? Maybe a small 10-minute talk, in case you're free sometime? In the meanwhile, I'll try to read up more about it.

[IshaGupta18]

Maybe we can come a little before the open call or something? I also wanted to let you know that I am working on CODAP and UI testing a little. And I also listed the flow of things in the integration PR https://github.com/publiclab/plots2/pull/5993#issuecomment-510356435

[jywarren]

I may be a bit busy this week at a conference, but I will give some hints here! Start by creating a new method in node_shared.rb, a copy of the existing graph method that can pattern match something similar like [simple-graph:/path/example.csv]

List the new method in this file right below the existing graph method:

https://github.com/publiclab/plots2/blob/75f3a7e23603c83fd0bfcd74cb38457f0e95fd86/app/helpers/application_helper.rb#L75-L90

Then upload a CSV in a wiki page to test on locally. You should be able to manually change the current default "graph" to "simple-graph" in the short code on your wiki, and see your new graph code generate. At first it'll be the same as the existing one. But if you make a new template like _simple-graph.html.erb, you'll be able to point your new method at that template instead.

Try opening a PR with the above elements. It doesn't have to be perfect but I can leave comments to help point out anything you've missed. This is a good working approach anyways as then everyone is on the same page and we can point at specific lines of code.

Just remember we don't have to make the entire system at once. We can do it piecemeal and it usually results in better modularization of code anyways!

[jywarren]

So you can start by just copying the new files and methods and I can help you modify and customize them as a next step!

[IshaGupta18]

Alright! Thanks a ton! I will go through it and implement all that I can and get back to you. Thanks again and good luck with the conference!

On Sun, Jul 14, 2019, 6:58 PM Jeffrey Warren notifications@github.com wrote:

I may be a bit busy this week at a conference, but I will give some hints here! Start by creating a new method in node_shared.rb, a copy of the existing graph method that can pattern match something similar like [simple-graph:/path/example.csv]

List the new method in this file right below the existing graph method:

https://github.com/publiclab/plots2/blob/75f3a7e23603c83fd0bfcd74cb38457f0e95fd86/app/helpers/application_helper.rb#L75-L90

Then upload a CSV in a wiki page to test on locally. You should be able to manually change the current default "graph" to "simple-graph" in the short code on your wiki, and see your new graph code generate. At first it'll be the same as the existing one. But if you make a new template like _simple-graph.html.erb, you'll be able to point your new method at that template instead.

Try opening a PR with the above elements. It doesn't have to be perfect but I can leave comments to help point out anything you've missed. This is a good working approach anyways as then everyone is on the same page and we can point at specific lines of code.

Just remember we don't have to make the entire system at once. We can do it piecemeal and it usually results in better modularization of code anyways!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ2E447C2QSG22BTPOLP7MSXVA5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ4FM2A#issuecomment-511202920, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXHQZ3BMUZ52LINYADCHUDP7MSXVANCNFSM4H5H5UNA .

[IshaGupta18]

Sure, let me work this out, and I'll let you know of what has been done and what complications I face :)

On Sun, Jul 14, 2019, 6:59 PM Jeffrey Warren notifications@github.com wrote:

So you can start by just copying the new files and methods and I can help you modify and customize them as a next step!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/simple-data-grapher/issues/65?email_source=notifications&email_token=AJXHQZ77LQSBCXN3PQRHS7TP7MSZ3A5CNFSM4H5H5UNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ4FNDQ#issuecomment-511202958, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXHQZ6IJYTU7I3F666OCQ3P7MSZ3ANCNFSM4H5H5UNA .