benloh commented 2 years ago

IMPORTANT: Merge #169 before merging this!

To Do

[x] Export as two files dataset_nodes.csv / dataset_edges.csv
[x] Combine "Vocabulary" and "Help" as "More"
[x] Add TOC for "More"
[x] ~~Collapsible Search/Node~~ -- Won't do, not necessary (maybe for future)
[x] Autocollapse Search/Node when Filter is opened for smaller screens
[x] Filter display in collapsible right panel

This implements #177.

Branch: dev-bl/export

This adds the ability to Export and download a CSV file to a local file.

Prototype Implementation

Currently this takes a preliminary approach to exporting. We can work out exactly how you want this to work.

To Test

git fetch; git checkout dev-bl/export
Start NetCreate with your own data, e.g. ./nc.js --dataset=junk
Go to http://localhost:3000
Log in if necessary
Click the "Help" tab
Click the "Export" button.
Set the download name.
Click "Save" to save the CSV file to your local hard drive.
Open the CSV in Excel

Where to put the Export button?

There is now an "Export" button in the "Help" tab.

I didn't want to add another tab because we already had so many tabs and on narrower windows the tabs do not flow well. We can revisit where to put the button

What is exported?

To keep things simple, we currently just export whatever is being drawn on the graph. Any nodes/edges that are filtered are not exported. Any nodes/edges that are highlighted ARE exported but there is no marker to indicate that they are highlighted.

We can add a second button "EXPORT FILTERED DATA" to "EXPORT DATA" if you think we need that distinction.

What is the export format?

Rather than exporting two separate files, I've put all the data in one file, something like this:

NODES
<node headers>
<nodes>

EDGES
<edge headers>
<edges>

It seems more convenient to be able to keep things in a single file. You can easily copy and paste to move them as needed.

If they should be exported as two separate files, let me know.

What is the CSV format?

Currently we're just building up the nodes and edges data based on the data in the database. The specific fields that are exported can easily be configured via the code. But right now there is no end-user customization available.

It's not clear we want to support end-user customization as that can get pretty complicated. The issue is that the data is not a simple flat table but consists of nested fields, so creating the index for the fields is something only an expert should do.

How to handle 'attributes' fields?

Our file format uses nested attributes fields for each node/edge record. I believe this came from whichever package we were originally importing the data from (Gephy?). e.g. a node record might look like this:

node = {
  id,
  label,
  attributes: {
    Node_Type,
    Extra Info,
    Notes,
  }
}

When exporting the data, we flatten out the data so that everything can fit within a single record. We do this by adding the 'attributes' tag to the field names, e.g. we have attributes:Node_Type, attributes:Extra Info, andattributes:Notes`, e.g.:

node = [
  id,
  label,
  attributes:Node_Type,
  attributes:Extra Info,
  attributes:Notes
]

Let me know if you need a different way of handling that data.

How are source and targets referenced in Edge nodes?

Currently, we're using numeric IDs to reference source and targets in edges. We can also add labels if needed, but that does get more complicated. Using ids seemed like the most efficient format.

Let me know if you need labels as well.

How to handle Commas, Quotes, and Special Characters?

To keep things simple, right now all fields are wrapped in double quotes ("") which should theoretically support commas in descriptions. We can strip out other characters if needed, but I wasn't sure how aggressive we should be.

The critical symbols are probably:

commas
double quotes -- depending on the application reading the final csv, we might be able to replace double quotes with single quotes, or otherwise encode them.
line feeds/carriage returns -- how do you want to preserve these? line feeds are the record delimiters in CSV.
control characters

I expect to tackle the encoding issues later.

How to set the default filename?

By default, the filename is '_export.csv'. We can use a different default filename if you prefer.

How to display date and time?

Both nodes and edges keep track of created and updated date and time information. We export the created and updated dates in UTC format. We can relatively easily select a different format if you prefer. Or of course we can remove dates altogether.

Thu, 11 Nov 2021 23:56:54 GMT

Any other issues?

kalanicraig commented 2 years ago

Aaaaaaah! Awesome. Joshua just tested an export and looked at it together.

EXPORT FORMAT AND NAMING

Two export files would be better, labeled _nodes.csv and _edges.csv . It’s less portable but will make for fewer errors in import to other network files, and most network programs will make two files for export, one for nodes and one for edges (or an XLSX file with two tabs, but two CSV files maps better to that model anyway.)

We should use the loki file name as the prefix for the csv export (eg Junk.loki = Junk_edges.csv)

I have a new network that needs some filtering so Ill test it thoroughly tomorrow with export and my kludgy import.

EXPORT CONTENT

The filtered export is also standard for Gephi, so while we’ll need to document that behavior, I agree with it 100%.

I also agree that end-user customization is not necessary. They can do that outside of Net.Create. I would add some of the admin stuff (eg last updated by, number of revisions) that’s in the loki file already.

Character escaping also seems logical. There’ll likely be other characters (@, asterisk, etc) but I need to look at how Gephi, Cytoscape and Statnet handle Unicode. I have a loki file in Japanese that would be a good test, so I’ll play with that tomorrow too.

IMPORT-EXPORT AND FILTER-HIGHLIGHT LOCATION

We have been debating. Joshua still wants pop ups (he is wrong; he says we are wrong 🤣). After some thought, I would propose:

Combine the “Help” and “Vocabulary” tab and label it “More”. Internal anchor links in a table of contents the top would make the potential endlessness of a “more” tab less scary We’d have to manage what’s actually stored there and what’s a link to a separate browser window, but that’s a tech-writing task, not a dev task.
The import/export buttons would be at the top of the More tab and would always be at the top above the table of contents.
Move the Filter tab to the very right hand side of the tab menu and make it a button that controls a panel that shows/hides from the right like the MEME evidence tab. This would behave differently than the current help tab in that it wouldn’t overlay the graph and the node/edge table.

Thoughts on the feasibility of item 3?

—Kalani

On Thu, Nov 18, 2021 at 8:45 PM benloh @.***> wrote:

[This is a work in progress...]

This implements #177 https://github.com/netcreateorg/netcreate-2018/issues/177.

Branch: dev-bl/export

This adds the ability to Export and download a CSV file to a local file. Prototype Implementation

Currently this takes a preliminary approach to exporting. We can work out exactly how you want this to work. To Test

git fetch; git checkout dev-bl/export

Start NetCreate with your own data, e.g. ./nc.js --dataset=junk

Go to http://localhost:3000

Log in if necessary

Click the "Help" tab

Click the "Export" button.

Set the download name.

Click "Save" to save the CSV file to your local hard drive.

Open the CSV in Excel

Where to put the Export button?

There is now an "Export" button in the "Help" tab.

I didn't want to add another tab because we already had so many tabs and on narrower windows the tabs do not flow well. We can revisit where to put the button What is exported?

To keep things simple, we currently just export whatever is being drawn on the graph. Any nodes/edges that are filtered are not exported. Any nodes/edges that are highlighted ARE exported but there is no marker to indicate that they are highlighted.

We can add a second button "EXPORT FILTERED DATA" to "EXPORT DATA" if you think we need that distinction. What is the export format?

Rather than exporting two separate files, I've put all the data in one file, something like this:

NODES
EDGES It seems more convenient to be able to keep things in a single file. You can easily copy and paste to move them as needed. If they should be exported as two separate files, let me know. What is the CSV format? Currently we're just building up the nodes and edges data based on the data in the database. The specific fields that are exported can easily be configured via the code. But right now there is no end-user customization available. It's not clear we want to support end-user customization as that can get pretty complicated. The issue is that the data is not a simple flat table but consists of nested fields, so creating the index for the fields is something only an expert should do. How are source and targets referenced in Edge nodes? Currently, we're using numeric IDs to reference source and targets in edges. We can also add labels if needed, but that does get more complicated. Using ids seemed like the most efficient format. Let me know if you need labels as well. How to handle Commas, Quotes, and Special Characters? To keep things simple, right now all fields are wrapped in double quotes ("") which should theoretically support commas in descriptions. We can strip out other characters if needed, but I wasn't sure how aggressive we should be. The critical symbols are probably: - commas - double quotes -- depending on the application reading the final csv, we might be able to replace double quotes with single quotes, or otherwise encode them. - line feeds/carriage returns -- how do you want to preserve these? line feeds are the record delimiters in CSV. - control characters I expect to tackle the encoding issues later. How to set the default filename? By default, the filename is '_export.csv'. We can use a different default filename if you prefer. Any other issues? ------------------------------ You can view, comment on, or merge this pull request online at: https://github.com/netcreateorg/netcreate-2018/pull/179 Commit Summary - e2c1bdf filter: Stop tracking netcreate-config.js. This is an autogenerated file that should not be in the repo. - 23ee166 filter: Move comment to the correct code block. - 7287873 filter: lint - disable complexity complaint. - a47e7ee filter: Rename "Filters" panel to "Highlight" - 15f84f2 filter: Add Filter tab. - 3288d1d filter: Add 'filterAction' to Filter components. - ac62e84 filter: Show different help text based on filter action. - 79a8bf8 filter: Disallow input if filter operator is not yet selected. - c8426b5 filters: Just use 'filteredTransparency' instead of also using 'isFilteredOut'. The parameters are redundant. - a97cce6 filters: Move 'componentWillUnmount' call to proper order (before custom methods) - 6bed8f2 filters: Combine Highlight and Filter into a single tab with a toggle button. - 3d43e0b filters: Update FILTERED_D3DATA when D3DATA is updated. This will trigger the graph redraw. - b6f67c8 filters: Only do graph redraw on FILTERED_D3DATA changes, not on D3DATA. D3DATA is the core data. We always only draw filtered data changes. - 60f9245 filter: Rewrite `m_FiltersApplyTo*` to handle either highlighting or filtering. - 45fc055 filter: Clarify filter behavior. - 4b0cf18 filter: Fix InfoPanel tab indices. - a87673e filter: Disable select filter value if NO_OP. - 5285d60 filter: Specify summary type. Remove summary field if no filters are set. - 59d36b3 filter: Lint/clean up formatting. - 26454d6 filter: Change FILTERED_D3DATA message to FILTEREDD3DATA app state. This is necessary for table updates to display current filtered data. - 2dde6a2 filter: Lint - 8b64258 filter: Improve NodeTable and EdgeTable layout. Tighten up spacing, decrease font size, adjust column size. - 1d30df4 filter: Remove edgeCount code, replaced by `degrees`, which is calculated via radius, which in turn is already based on edge count. - a246604 filter: Lint/doc. - 32f5196 filter: Remove stray edgeCount refernces. - 897f553 filter: Show highlighted/filtered state in NodeTable. - 2588af7 filter: Show highlighted/filtered state in EdgeTable - ced9393 filter: Doc - 6e491d5 filter: Prevent "Enter" on Filters from submitting form. Addresses #172. - 2bb2ac5 filter: Use 'visibilioty' to hide graph when not logged in. Don't use React's 'hidden' because that prevents the graph size from being calculated. Addresses #171 - 531078d filter: lint - 7812e9b filter: Add ZOOM_PAN_RESET call. - 5b4b7ab export: Add "Export" button to Help tab. - de464f8 export: Handle "Export" click - cdfb27c export: Add export-logic to handle export request. File Changes (18 files ) - *D* build/app/assets/netcreate-config.js (9) - *M* build/app/view/netcreate/NetCreate.jsx (17) - *M* build/app/view/netcreate/components/EdgeTable.jsx (116) - *M* build/app/view/netcreate/components/Help.jsx (14) - *M* build/app/view/netcreate/components/InfoPanel.jsx (8) - *M* build/app/view/netcreate/components/NetGraph.jsx (1) - *M* build/app/view/netcreate/components/NodeTable.jsx (210) - *M* build/app/view/netcreate/components/d3-simplenetgraph.js (56) - *M* build/app/view/netcreate/components/filter/FilterEnums.js (5) - *M* build/app/view/netcreate/components/filter/FilterGroup.jsx (8) - *M* build/app/view/netcreate/components/filter/FilterGroupProperties.jsx (9) - *M* build/app/view/netcreate/components/filter/FiltersPanel.jsx (44) - *M* build/app/view/netcreate/components/filter/NumberFilter.js (17) - *M* build/app/view/netcreate/components/filter/SelectFilter.js (26) - *M* build/app/view/netcreate/components/filter/StringFilter.jsx (19) - *A* build/app/view/netcreate/export-logic.js (199) - *M* build/app/view/netcreate/filter-logic.js (198) - *M* build/app/view/netcreate/nc-logic.js (9) Patch Links: - https://github.com/netcreateorg/netcreate-2018/pull/179.patch - https://github.com/netcreateorg/netcreate-2018/pull/179.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub , or unsubscribe .

benloh commented 2 years ago

Thanks!

_edges.csv and _nodes.csv

The natural way to do this would be to split "Export Data" into two buttons: "Export Nodes" and "Export Edges". Is that what you want? We might be able to zip them, but that seems like it defeats the purpose.

I would add some of the admin stuff (eg last updated by, number of revisions) that’s in the loki file already.

We're already exporting the created and updated dates. I don't think we're updating the revisions key. Should we be?

filters right panel

This should be doable. Though keep in mind the filters do take up a fair amount of horizontal space, so the panel would be pretty wide. So when it's open you would only have a small sliver of graph showing.

I suppose an alternative solution would be to replace the Search/Node Edit panel on the left? Or perhaps even collapse the Search/Node panel when Filters is open?

Either way, the vertical layout means we would have to show nodes on top and edges on bottom, so you might have to scroll to get to all of the fields. One advantage of the current layout is that you can see ALL the fields.

jdanish commented 2 years ago

I think having the option to collapse search/node would be great. If it's not too much harder, my vote would be make it optional, but auto-collapses if the screen resolution is below a certain number? On my laptop at my resolution there is plenty of room and I wouldn't bother. So its for the students with crappy old laptops that we need it.

As for the rest I defer to Kalani, though one question: when you do the import, will the two files model work? Or will that cause problems? I'm OK with either individual exports or the zip file as long as it doesn't impact import dramatically.

jdanish commented 2 years ago

For reference, here is Gephi with a similar layout.

Our student users tend to focus more on the node attributes thus far, then the table view, so having the tables crunched between the side panels makes sense to me even though the data guy in me likes how Gephi is giving lots of table space.

Otherwise I think this works well especially if we have easy collapse and restore as in meme.

benloh commented 2 years ago

Well, now's the time to change the overall layout if you like the wide tables. It should be easy enough to do given how everything is just a component.

jdanish commented 2 years ago

I think I like / prefer our approach to the narrower table and more attention to the nodes and filters. @kalanicraig?

benloh commented 2 years ago

two import files If we're keeping things as two separate files, then I imagine the import process will be:

Click Import nodes

Click Import edges

Click Process Import (or something like that)

I suppose we could just process as soon as edges come in. I think making it a zip file will probably complicate things: you'd have to take the extra step of zipping before you could upload.

jdanish commented 2 years ago

Yeah, so two separate exports makes sense.

kalanicraig commented 2 years ago

Two buttons (export nodes and export edges) would get us that (and allow for easier export of edges if no new nodes have been introduced)

On Nov 19, 2021, at 1:37 PM, Joshua Danish @.***> wrote:

Yeah, so two separate exports makes sense.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-974313250, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NEJJFT4P4V2CN2RNC3UM2KORANCNFSM5ILBCEHA.

benloh commented 2 years ago

Also, @jdanish @kalanicraig just making sure you've checked the exported edge documents and that the use of ids for source and targets is OK? No need for source/target labels?

kalanicraig commented 2 years ago

Yes, IDs for edges rather than labels is my preferred behavior.

On Nov 19, 2021, at 1:49 PM, benloh @.***> wrote:

Also, @jdanish https://github.com/jdanish @kalanicraig https://github.com/kalanicraig just making sure you've checked the exported edge documents and that the use of ids for source and targets is OK?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-974321661, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NETLZOOBLRRBSB7SYTUM2L5DANCNFSM5ILBCEHA.

kalanicraig commented 2 years ago

I am torn on the wide vs narrow tables. My inclination would be: what is easier? If the side panels are collapsible, then it’s really user choice, and so I’d lean toward collapsible side panels with the table data filling the space between the side panels (hidden or displayed).

On Nov 19, 2021, at 12:58 PM, benloh @.***> wrote:

Well, now's the time to change the overall layout if you like the wide tables. It should be easy enough to do given how everything is just a component.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-974287015, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NFKR5TITNEHXMKYZL3UM2F5XANCNFSM5ILBCEHA.

jdanish commented 2 years ago

Then we have consensus! The table should stay narrower between the two side panels (if they are open).

Thanks!

kalanicraig commented 2 years ago

Following this up because we are having meatspace conversations after each email:

To clarify: my suggestion here is that we keep the node/edge tables in their current position, with the node-display left-hand panel and the filter-display right-hand panel prioritized when they are set to visible.

On the filter display: there are currently two columns of filter selections. Can we set those up so that they are stacked top to bottom rather than left to right to help with some of the width issues?

On Nov 19, 2021, at 1:51 PM, Kalani Craig @.***> wrote:

I am torn on the wide vs narrow tables. My inclination would be: what is easier? If the side panels are collapsible, then it’s really user choice, and so I’d lean toward collapsible side panels with the table data filling the space between the side panels (hidden or displayed).

On Nov 19, 2021, at 12:58 PM, benloh @. @.>> wrote:

Well, now's the time to change the overall layout if you like the wide tables. It should be easy enough to do given how everything is just a component.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-974287015, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NFKR5TITNEHXMKYZL3UM2F5XANCNFSM5ILBCEHA.

benloh commented 2 years ago

Great!

two columns of filter selections

If I'm understanding you correctly, I believe that's what I was suggesting. e.g. something like this:

NODES FILTERS
Label: [ contains ] [ theo]
Type: [ ][ ]
...
Transparency:[ ]

EDGE FILTERS
Source: [ ] [ ] 
Type: [ ] [ ] 
Target: [ ] [ ] 
...
Transparency:[ ]

kalanicraig commented 2 years ago

Yes, that’s exactly what I was thinking in terms of stacking.

Hooray!

On Nov 19, 2021, at 2:04 PM, benloh @.***> wrote:

Great!

two columns of filter selections

If I'm understanding you correctly, I believe that's what I was suggesting. e.g. something like this:

NODES FILTERS Label: [ contains ] [ theo] Type: [ ][ ] ... Transparency:[ ]

EDGE FILTERS Source: [ ] [ ] Type: [ ] [ ] Target: [ ] [ ] ... Transparency:[ ]

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-974331905, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NCPA7NDJS622HW6XJDUM2NVFANCNFSM5ILBCEHA.

jdanish commented 2 years ago

@benloh FYI: I know you didn't ask for testing, but we looked and it looks great. However, the node / edge tables are definitely not consistently updating in response to filter changes, particularly changes between filtering and highlighting, the way the graph is. Just letting you know in case you haven't been looking. Of course, I can also wait for an official commit and file issues if that is easier.

benloh commented 2 years ago

@jdanish So for the node and edge tables, I assume that if an edge or node is filtered out (removed) then you don't want them to appear in the edge table at all? Or was my approach of using red to indicate removed status correct?

One related issue here is how to handle the export: when you export, would you expect the red removed items to be exported as well? Or is it clearer to simply remove them altogether from the table? So the exports will reflect the pared down table?

Un-highlighted nodes/edges are made transparent according to the transparency setting of the filter. And exports ignore highlight state.

kalanicraig commented 2 years ago

The red display of filtered nodes in-program is great.

For the export, we want to filter them out entirely to make layout in an external network analysis tool easier. We can always unfilter and reexport.

We demoed today for Ann (our IU Network Institute partner) and she’s super excited.

—Kalani

On Dec 2, 2021, at 6:38 PM, benloh @.***> wrote:

@jdanish So for the node and edge tables, I assume that if an edge or node is filtered out (removed) then you don't want them to appear in the edge table at all? Or was my approach of using red to indicate removed status correct?

One related issue here is how to handle the export: when you export, would you expect the red removed items to be exported as well? Or is it clearer to simply remove them altogether from the table? So the exports will reflect the pared down table?

Un-highlighted nodes/edges are made transparent according to the transparency setting of the filter. And exports ignore highlight state.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

benloh commented 2 years ago

@jdanish

However, the node / edge tables are definitely not consistently updating in response to filter changes, particularly changes between filtering and highlighting, the way the graph is.

This should now be fixed.

jdanish commented 2 years ago

Looks good! Thanks!

benloh commented 2 years ago

Special Characters

Double quotes -- escaped (by using two double quotes next to each other: "")
Line feeds inside of fields (e.g. "Significance") -- supported. they do not trigger a new record.
Single quotes -- supported
Emojis -- supported
Special language characters -- supported (e.g. Chinese, latin)

benloh commented 2 years ago

@jdanish I think the Export feature is basically done and working. The main question is whether you still think we need to add the ability to expand/collapse the search/edit node right sidebar. We probably do need to do some work to make it better reflow when it's reduced in width. But do we still need a button to expand and collapse the whole area?

I hesitate mostly because it doesn't seem necessary with the current layout (though I haven't tested on very small screens), eats up more vertical space, and adds another widget that you have to manage as you work.

jdanish commented 2 years ago

Awesome. Kalani is going to test import on that to verify it works well.

In tinkering on my laptop using Chrome's resizing, it looks like if someone is using a chromebook would have an OK experience, but we'd need some tightening of how the reflow works. On an iPad pro, though, it'd be too hard to see. I'll defer to Kalani on who we think the users are? From that I'd be OK with just cleaning up the reflow slightly, and assuming that if someone is using an iPad, they need to close the filter to really continue navigating and that is AOK. But @kalanicraig?

benloh commented 2 years ago

The Node Editor/Search area definitely needs some rework when minimized. I think the question is whether we need to implement a whole widget to allow explicit control of the collapse/expansion.

On Sat, Dec 4, 2021 at 5:18 AM Joshua Danish @.***> wrote:

Awesome. Kalani is going to test import on that to verify it works well.

In tinkering on my laptop using Chrome's resizing, it looks like if someone is using a chromebook would have an OK experience, but we'd need some tightening of how the reflow works. On an iPad pro, though, it'd be too hard to see. I'll defer to Kalani on who we think the users are? From that I'd be OK with just cleaning up the reflow slightly, and assuming that if someone is using an iPad, they need to close the filter to really continue navigating and that is AOK. But @kalanicraig https://github.com/kalanicraig?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-986026151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKWRMKE5DWFT337EE5O4NTUPIILFANCNFSM5ILBCEHA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jdanish commented 2 years ago

We voted (Kalani and I) and agree no need to make it collapsible. Kalani tested on an iPad pro and it was fine despite slight awkwardness.

(from my iPhone)

Joshua Danish http://www.joshuadanish.com

On Dec 4, 2021, at 12:46 PM, benloh @.***> wrote:

The Node Editor/Search area definitely needs some rework when minimized. I think the question is whether we need to implement a whole widget to allow explicit control of the collapse/expansion.

On Sat, Dec 4, 2021 at 5:18 AM Joshua Danish @.***> wrote:

Awesome. Kalani is going to test import on that to verify it works well.

In tinkering on my laptop using Chrome's resizing, it looks like if someone is using a chromebook would have an OK experience, but we'd need some tightening of how the reflow works. On an iPad pro, though, it'd be too hard to see. I'll defer to Kalani on who we think the users are? From that I'd be OK with just cleaning up the reflow slightly, and assuming that if someone is using an iPad, they need to close the filter to really continue navigating and that is AOK. But @kalanicraig https://github.com/kalanicraig?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-986026151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKWRMKE5DWFT337EE5O4NTUPIILFANCNFSM5ILBCEHA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

benloh commented 2 years ago

@jdanish @kalanicraig Alright, I think the layout issues have mostly been fixed now. The whole app can probably use a relayout/cleanup, especially for narrower screens, but we'll save that for the end.

Fixes:

FILTER button no longer crows "More..." button
Node and Edge Table resizer correct position has been restored (the drag position was off by 40px)
The Node and Edge Table heights are now correct (the scrollbar would extend beyond the bottom of the visible area by 40px).
The Node and Edge Table headers now stay fixed as the table body scrolls.
The Search label "Type to search or add a node:" no longer crowds the "Add New Node" button.

Please give it a whirl and if things look good, let's merge this so we can move onto import.

benloh commented 2 years ago

Kalani wrote:

I tested export several times and the only thing I see happening is with the ID label in the nodes and edges export. The nodes “id” field needs to be uppercase “ID” and the ID field for the edges should be at the end of the export column list. The Source and Target ID fields in the edge table need to be labeled with the sentence case.

benloh commented 2 years ago

@kalanicraig Some questions:

The nodes “id” field needs to be uppercase “ID” Do you mean the header needs to say "ID"?

the ID field for the edges should be at the end of the export column list Here you're saying that for exported edges, you want the ID field to be the last item in the list? e.g. For ID = 595, instead of this:
// OLD
"595","The Briber","","","","0","Fri, 04 Sep 2020 13:53:39 GMT",
...you want this:
// NEW
"The Briber","","","","0","Fri, 04 Sep 2020 13:53:39 GMT","595"
The Source and Target ID fields in the edge table need to be labeled with the sentence case. I'm not sure what you're referring to here. Are you saying in the exported edges csv the header should read Source and Target instead of the current source and target?

benloh commented 2 years ago

@kalanicraig One more question/issue on exports: For edges I believe you had said that you wanted to be able to specify the source and target nodes via the node labels rather than the node ID numbers. The problem is that we HAVE to use the ID numbers because the labels are not guaranteed to be unique, e.g. you can have two nodes named "Alexandria". If you want to be able link via labels only then, we need to:

Always check to make sure node labels are unique (e.g. when editing a node, if someone enters a duplicate label, we would need to prevent that and tell the user to enter something unique).
Change the edges export format to either also include labels with id numbers, or remove the id numbers.

This seems doable so long as it matches your workflow. I don't have a sense of the capabilities and limitations of the tools you're using outside of NetCreate to manipulate the data.

So should we remove IDs from the edges and just use Source and Target labels AND change the editor to not allow duplicate node labels?

kalanicraig commented 2 years ago

Item 2 would be the preferred option so that we can do easy lookups.

We still want the edge ID to go with the edge export, but yes, the edge ID should be at the end of the line so that less technical folks are focused on the Source and Target IDs as the important key-value references for the edge table.

ID (all caps) and Label (sentence case) in the nodes field, and “Source” and “Target” in the edges field (with IDs rather than labels in those columns) are the key pieces of Gephi and Cytoscape’s imports. That’s where I fail alllllllll the time, because I’ve forgotten to label them correctly. If we can fix those on import, it’ll be a nice QoL even for technical folks. I usually title the Label columns “Source Label” and “Target Label” if I want an explicit human-readable value to go with the IDs in the edge table.

On Dec 22, 2021, at 1:25 PM, benloh @.***> wrote:

@kalanicraig https://github.com/kalanicraig One more question/issue on exports: For edges I believe you had said that you wanted to be able to specify the source and target nodes via the node labels rather than the node ID numbers. The problem is that we HAVE to use the ID numbers because the labels are not guaranteed to be unique, e.g. you can have two nodes named "Alexandria". If you want to be able link via labels only then, we need to:

Always check to make sure node labels are unique (e.g. when editing a node, if someone enters a duplicate label, we would need to prevent that and tell the user to enter something unique). Change the edges export format to either also include labels with id numbers, or remove the id numbers. This seems doable so long as it matches your workflow. I don't have a sense of the capabilities and limitations of the tools you're using outside of NetCreate to manipulate the data. — Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-999779008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NH324OZS4FZNED5CK3USIJZXANCNFSM5ILBCEHA. You are receiving this because you were mentioned.

benloh commented 2 years ago

@kalanicraig I think the challenge here is that you're talking about three different types of labeling:

matching our internal code representation
matching Gephi/Cytoscape labeling
making the label human readable.

For example, if we edges with "Source Label", does Gephi/Cytoscape recognize that? Don't they need it to be "Source"?

Do you need the ability to designate different label mappings? Or can we keep things simpler and just choose one? (e.g. only have a Gephi/Cytoscape, not also a Human Readable?

Also, I wanted to confirm that the attributes and meta information mappings are working? I wasn't sure how Gephi/Cytoscape handle those extra fields?

kalanicraig commented 2 years ago

Right. So, given that list, I’d privilege interoperability between Gephi and Net.Create, with a side of human readability, over full interoperability with all possible systems.

Gephi node tables require: ID: numeric only Label: Any

Gephi’s edge table requires: Source: numeric ID from node table Target: numeric ID from node table

Gephi prefers a “Type” column in edge import that is “Directed” or “Undirected” but there’s a batch setting in the import process itself that supports users in choosing directed/undirected

Everything else that Gephi imports comes in or goes out as an attribute. If we prioritize items 1 and 2, then I imagine it would look something like this: We highly recommend exporting from an existing database and using the output as a guide for import, with specific limits that require: NodeID matching for Source/Target EdgeID maintenance for existing edges that need to be modified in some way On import, require: NODE TABLE IMPORT: Numeric-only “ID" and “Label" columns All other columns on import are matched to the attributes in the template and user gets big giant warning that any columns that don’t match existing attribute values in the template won’t be imported. EDGE TABLE IMPORT: “Source” and “Target” numericID columns that relate to entry in nodeID table EdgeID column with NetCreate’s edgeID value for an edge that already exists and blank for new edges Recommended SourceLabel and TargetLabel which Net.Create import will ignore so that user can spot-check human-readable labels in import data against Net.Create import results All other columns on import are matched to the attributes in the template and user gets big giant warning that any columns that don’t match existing attribute values in the template won’t be imported.

It’s not super human readable, but it’s got enough there that it would function with a little documentation (and now we have some of that in nascent form here to adapt)

On Jan 17, 2022, at 2:13 PM, benloh @.***> wrote:

@kalanicraig https://github.com/kalanicraig I think the challenge here is that you're talking about three different types of labeling:

matching our internal code representation matching Gephi/Cytoscape labeling making the label human readable. For example, if we edges with "Source Label", does Gephi/Cytoscape recognize that? Don't they need it to be "Source"?

Do you need the ability to designate different label mappings? Or can we keep things simpler and just choose one? (e.g. only have a Gephi/Cytoscape, not also a Human Readable?

— Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-1014825727, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NBKH4NTILWZ5R56NTLUWRS6RANCNFSM5ILBCEHA. You are receiving this because you were mentioned.

benloh commented 2 years ago

@kalanicraig I think we probably need to make some of this editable via the template too.

Can you send me a prototypical Gephi export? I know you've sent one before, but let's start fresh with an real-world use case.

I'm especially confused by the Edge Type. Should we be representing that internally as well?

Popping up a level, it kind of seems like the idea case is that are able to import and export in Gephi format rather than some Net.Create proprietary format?

benloh commented 2 years ago

@kalanicraig @jdanish This may have gotten lost in the slew of updates/emails I was sending:

I think we probably need to make some of this editable via the template too.

Can you send me a prototypical Gephi export? I know you've sent one before, but let's start fresh with an real-world use case. I'd like to see what the raw file looks like (I'm assuming it's csv).

I'm especially confused by the Edge Type field. Should we be representing that internally as well?

Popping up a level, it kind of seems like the ideal case is that we are able to import and export in Gephi format rather than some Net.Create proprietary format?

kalanicraig commented 2 years ago

Hi! I'm attaching an Excel file that was used to import into Gephi as well as the export CSVs that came from an export of that network from Gephi. Bonus points for mixed character sets and some double quotes.

I also used this Excel file to concatenate JSON lines for the Nodes and Edges but that document is in a colleague's OneDrive and I can't get to it right now.

On Wed, Jan 19, 2022 at 12:58 PM benloh @.***> wrote:

@kalanicraig https://github.com/kalanicraig @jdanish https://github.com/jdanish This may have gotten lost in the slew of updates/emails I was sending:

I think we probably need to make some of this editable via the template too.

Can you send me a prototypical Gephi export? I know you've sent one before, but let's start fresh with an real-world use case. I'd like to see what the raw file looks like (I'm assuming it's csv).

I'm especially confused by the Edge Type field. Should we be representing that internally as well?

Popping up a level, it kind of seems like the ideal case is that we are able to import and export in Gephi format rather than some Net.Create proprietary format?

— Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/pull/179#issuecomment-1016722190, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NAX7SCHKUGNY3TBTMLUW33UJANCNFSM5ILBCEHA . You are receiving this because you were mentioned.Message ID: @.***>

benloh commented 2 years ago

@kalanicraig Thanks! Unfortunately replying to github doesn't attach the file. You'll have to eithe log into github and attach it there or you can just email me directly. Thanks!

benloh commented 2 years ago

Merging export for now.
This has a hacked in override for defining headers, e.g. id is exported as ID. This will be replaced with a template definition with #175.

netcreateorg / netcreate-2018

Export #179

IMPORTANT: Merge #169 before merging this!

To Do

Prototype Implementation

To Test

Where to put the Export button?

What is exported?

What is the export format?

What is the CSV format?

How to handle 'attributes' fields?

How are source and targets referenced in Edge nodes?

How to handle Commas, Quotes, and Special Characters?

How to set the default filename?

How to display date and time?

Special Characters