Fixes/tweaks to existing features

gabrielkg commented 3 years ago

An ongoing list of fixes/tweaks to existing features.

Comments with open items :construction:

2021Jan22 gabrielkg
2021Jan27 gabrielkg. (and 2 following comments)
2021Feb22 Don-Isdale
2021Aug26 Don-Isdale
2022Sep16

Previously completed:

Explorer tab

[x] Remove Filters box (d33fb80)
[x] Search box always visible by default, within Datasets box (345aa349)
[x] Search all datasets below including “All Datasets”
[x] Simple text matching on Block or Dataset name (345aa349)
- [x] matching on Block name (0a75e6d3)
- [x] input string separated by spaces translates to AND query, eg: "IWGSC 90k" should match "IWGSC" AND "90k"
- - [x] split on white-space and match all texts given (eadea818)
[x] Upload new data button should be at the top of the Datasets box (a253227)
[x] Datasources open by default (fa61b93a)
[x] Don’t show blue text https___plantinformatics_io (d93136e5)
[x] No + button in Datasets box (e556392)
[x] In Datasources box: change "New" -> "Connect new datasource" (and add an icon similar to the upload icon if something suitable) (d93136e5)
[x] (Priority 3) When first requesting list of datasets to populate the Dataset explorer, default message is "No datasets found!" before it receives... Should really say "Fetching datasets..." unless the returned list is found to be empty. (0379b6fd)

Feature search

[x] Input box doesn’t allow newline with enter (converts a space to newline) (c4044cf6)
[x] Axis ticks not shown with feature search if split axis (maybe should be a triangle on the side of axis? - show feature search result with triangle at all times)
- [x] feature search result with triangle (fc77bf3b), styling dcbfd40d, add fill 053aeb3b.
- [x] show also when split axis is open (228fd35e) the horizontal tick (path.horizTick) should not be shown; always show a triangle instead. this will be outside the axis if split, on the left side, pointing toward the axis (i.e. right). (not related to the axis tracks appearing as triangles if sparse)
[x] (Priority 5) Add selected features to search box button should append rather than replace (10d0f536) (de1b54fb : retain the appended value; excise selectedFeatures duplicates.)

View tab

[x] Publish mode -> "Show Zoom/Reset buttons" (e556392)
[x] Remove "Add data from the explorer tab" button (581a0d3d)
[x] (Priority 2) Chart options should be collapsed by default (61391e7e)

Right hand panel Features tab

[x] Hide the filters box (e556392)

Visualisation

[x] Flip Region redraws paths immediately but red dots have a slow transition. Remove transition on red dots. (042f85b2)
- [x] Also issue #202.
[x] No ticks/triangles for blocks with paths: false, either split axis or not (73bbf4d3)
[x] Show orientation of features in split axis with small triangle at end of feature, with point at the final position of the feature (a4d766aa, 77c7339e, 55069b25, 85aa8f61; f1521e4e : flag out for now))
[x] Assign block colour on initial load, before any features are received (b33829a8)
[x] axis tracks : when zooming in, track rectangles moving out of scope transition to the top & bottom edge of the split axis, piling up there until zooming ends, then they are removed. Should remove in the normal time-frame for the transition. (0d03e023)

Metadata

[x] Issue #222 - Editing no long works (be584c12 : fixes 2nd&3rd points, 52a45fb9 : refresh viewer when dataset change, etc, c29cf09c : use fork, make jsonDidChange() not an observer.)

General

[x] Clicking axis title, or clicking dataset in View tab Displayed Data list, should switch right panel to Dataset tab and show metadata (43d63a52 : connect selectBlock() to selectDataset; 5e4559e9 : showMenu() : send selectBlock)

branch feature/endPosition 2021Aug27 - , from meeting 2021Aug26, plus items found in progress.

[x] [1-3H/2H] axis flip doesn't yet apply to end position. 97b322cb
[x] [1-2H/1H] brush : filter does not yet check end position : should include if [start,end] overlaps the brush (thumb). 51a74608
[x] [1-3H/1H] ordering of blocks in the explorer : getting things really out of order on eucalyptus for some reason (dev with eucalyptus as secondary) 1e07c4cf: dataset explorer : sort blocks by name
- [x] [1-2H/1H] To handle cases like this : Scaffold_1 Scaffold_10 Scaffold_11 Scaffold_2 ... we can use : http://www.davekoelle.com/alphanum.html https://github.com/cblanc/koelle-sort 85a7a56e: dataset explorer : use natural sort for blocks
- [x] [1-2H/2H] the above works for All Datasets; apply same change to the list of chromosomes in eg: Genomes tab 27d73899: dataset explorer : apply the scope/block sort used in All Datasets to the list of chromosomes in eg: Genomes tab
[x] [1H/1H] switch to dataset tab only when brush changes selected dataset. 5e0765c7
[x] [1H/1.5H] feature table highlight : handle . in feature name; same for path table. db3d2ea1: handle . in feature name : convert to _ for css class name
[x] [1-2H/1H] paths table : exceptions caused by mismatch in # headings a9177229 : paths table : fix exception caused by extra table.
[x] [1-2H/1H] support for citations files - .cff (bioarxiv) 4368ad41: Add CITATION.cff
[x] [2H/2.5H] user doc on meta types, e.g. meta.shortName : location and effect. a332c34b: add a description of the dataset.meta values in Pretzel JSON https://github.com/plantinformatics/pretzel/blob/a332c34b/doc/notes/dataset_meta.md
completed 3 items in #233 / Issues
[x] [1H/0.5H] sequence search : allow Ns in the sequence input. 89dac8c1

gabrielkg commented 3 years ago

[ ] Current search functionality is only accessible with Advanced option
[ ] Needs some View tab options moved to an Advanced options sub-list which is collapsed by default (needs discussion)
[ ] Clarification on meaning of some of the options (needs discussion)

Deferred for later consideration:

[ ] Issue #183 - does not return all expected hits (DEFERRED)
[ ] Add configuration to Features table to be like Paths table (taking up the expanded width) (DEFERRED)
[ ] Spacing of large features (eg: scaffolds in AGP) can be unpredictable. In following example, when sufficiently zoomed, x-position moves to the right, almost overlapping the next block, with whitespace to the left of it. Second image is single zoom in step from the first image.

gabrielkg commented 3 years ago

[ ] Mouseover tooltip on features is placed in the centre of the feature, which can be off the screen in the case of large features or if zooming in very close. Should be next to mouse position. (This was investigated before and appears to be difficult.)

gabrielkg commented 3 years ago

[ ] When split axis, for some blocks, features in the view keep transitioning to the top/bottom of the axis (as if leaving the view). Likely due to duplicate feature names. This was addressed in an earlier commit but seems to have returned. Though this is now happening with IWGSC HC genes v1.1, which shouldn't have duplicate feature names... While it doesn't happen for 90k markers, which is expected to have duplicates.

gabrielkg commented 3 years ago

[ ] There are cases where no requests for paths is made. Eg: load some blocks against IWGSC reference, zoom in, then load a genetic map and reset zooms. See image for example datasets.

gabrielkg commented 3 years ago

[x] Consistency of functionality on panel containers in the left panel. Eg: each should have a collapse/open button in top right (like Datasources has). (5fc10d18)

Don-Isdale commented 3 years ago

Tested OK with v2.5.0 [ edit : but it is necessary to zoom in until there are 0 paths to see the issue ]

* [ ]  There are cases where no requests for paths is made. Eg: load some blocks against IWGSC reference, zoom in, then load a genetic map and reset zooms. See image for example datasets.
  ![p3](https://user-images.githubusercontent.com/20571319/105946709-e4e07c00-60bb-11eb-82a0-f4d48c420659.jpg)

Screenshot from 2021-02-22 18-54-19

This issue only occurred when zoomed in so that there were 0 paths.

[x] Addressed in bc16213a.

Don-Isdale commented 3 years ago

This was found to be due to duplication of features in the JSON which was uploaded. This led to a discussion about data checking by upload, and concluded that if the upload JSON contains feature duplication of (name, start, end), then display a message to the user in the upload panel. May be preferred to complete the upload as the user may have intended to split feature data into 2 parallel features. In that case the trackKeyFn can use the feature db ID for uniqueness.

[ ] check for duplication of feature (name, start, end) in upload

* [ ]  When split axis, for some blocks, features in the view keep transitioning to the top/bottom of the axis (as if leaving the view). Likely due to duplicate feature names. This was addressed in an earlier commit but seems to have returned. Though this is now happening with IWGSC HC genes v1.1, which shouldn't have duplicate feature names... While it doesn't happen for 90k markers, which is expected to have duplicates.

Don-Isdale commented 3 years ago

from meeting 2021Aug26 :

look at cases where Pretzel has forward functionality but not undo : e.g. multiple backends - no disconnect

:zzz: [1-4H] dataset explorer : when viewing secondary data source in Genome tab, only one dataset (Exome_SNP) appears in each scope ; this is only seen so far local -> dev : Triticum_aestivum_IWGSC_RefSeq_v1.0 (2.0 is OK - 2 datasets) main->dev and dev-> main are OK

from meeting 2021Sep16 :

[x] [1-2H/2.5] case in/sensitive search filter in dataset explorer 8f1b3a41: dataset explorer search filter : toggle for case in/sensitive, and and/or
- [x] [0.5H/1H] change toggle text : case in/sensitve match all / any 83eb29a9: dataset explorer search filter toggles : change labels be more textual fb27f771 : change match button default from any to all 1763baf1 : initialise view history if none define yet in this browser.

from meeting 2021Sep21 :

[x] [2-3H/5H] dataset explorer tab for recent / favourite blocks (datasets) 231a2d3b, 01dc6c2a, 852a66de : default to not filter blocks by history; update when controls change. fe89ac00 : Change button label Normal to All 5e35a0c4, 4b138cc3 : dataset explorer : apply view history filter to the remaining tabs

[x] [/2.5H] fix : brush after feature search was getting exception. dbcf63bb

[x] [/1H] dataset explorer search input : button to clear input. 2edada1d : enable cancel button in Chrome

[x] [1-2H/~2H] axis-1d.js : blockColour() : allocate colourSlotsUsed per dataset 78461516: assign colour slots to datasets instead of blocks

[x] [2-3H/4H] option for Feature Search by alias. 1e5108ef
- [x] b4d1128c : Feature search results : indicate if the feature was matched via an alias.
- [x] 6239a172 : features search results : add click action on feature count to display feature list
- [x] 82578ea0 : feature search results : in place of the list of aliases, list the search names with aliases. 2021Nov08

[ ] [2-6H] Chrome svg export : no paths visible (should be both direct & alias), and axis drop zones appear as black rectangles
[x] [1-2H/3H] feature name -> id for circle class to handle '.' axis circle : change element id to feature id, to handle punctuation in feature names. 7f881acb

2021Dec15

[x] [/1H] feature search : show aliases search result label only if aliases .length. c36a4f5f
[x] [/2H] replace the Search Server URL text in feature search with server name in the same format as in explorer, with colour border. 0f7e662f
- [x] [/2H] replace use of class col-xs-12 with panel-section. 40d43948
- [x] [/0.2H] feature search : drop Search Server, which is now shown between the two search sections. 4592d2ce

2022Jan24

[ ] [2-6H] axis-tracks shows an exit transition when toggling off the visibility of blocks (including QTLs); in this case no transition is required - just disappear (or could transition width to 0; when becoming visible it transitions the width from 0.
[x] [/0.5H] feature triangles ticks / spanning line : use feature colour (QTLs) when enabled. f3509658

2022Feb24

[ ] [2-6H] feature table : request all (within brush), similar to request-all in paths table. (possibly duplicate)

2022Mar16

[ ] [2-4H] check that password reset is case-insensitive - should match email in db regardless of case.

2022May03

[x] [1-2H] axis end point does not contain feature end, e.g. feature [124, 124.2], axis ends at 124.0 tested OK Jun01 (0722c903)

2022May18

[ ] [1-2H] dataset Meta display is not refreshed by upload, click on another dataset and back again updates Meta display

2022May25

[x] [/2H] feature search : handle absence of featuresAliases. f6def519
[x] [/2H] display features & paths when they are at the edge of the domain. 46a3a708

May26

[x] [/1H] QTL spreadsheet upload : don't set .value_0 to []. fd9909c2

from meeting 2022May31 :

[x] [1-3H/2H] features table does not yet support duplicate feature names. bb0c6d1a
discussed : spreadsheet column instead of Metadata worksheet, i.e. feature.values.namespace could override block.namespace ? Currently block.namespace is used in collating aliases in pathsAliases(), not used in pathsDirect(). Using feature.values.namespace would complicate the query.
[ ] [1-3H] direct paths : can check that namespaces of the 2 blocks match on the frontend, reducing un-productive requests

spreadsheet upload of :

- aliases [4-12H] aliases are public; only the feature name information is exposed, not location etc. aliases are not grouped in blocks, so perhaps delete them individually, using the same data as create
- reference [2-4H]

[ ] [1-2H] features search via alias : aliases are public, so test if features in non-readable datasets are included;

2022Jun23, from Pretzel Chats

check : when switching between blast results tabs, Pretzel would originally add/remove axes that were displayed to display the blast search results

2022Aug08

[x] [/3H] auth.js : featureSearch() : authentication to secondary server fails and gets CORS error if featureNames is too long; rather than truncate at paramLimit, put longer data in POST body instead of GET URL 09f6fb8a. feature search : when many feature names, use POST instead of GET.

2022Aug09

show these upload errors in GUI : Error: Dataset name "... superscaffolds" is already in use Error: Authorization Required test case : upload dataset, don't make public, upload again (probably as another user) GUI currently shows "updating database"

Don-Isdale commented 2 years ago

2022Sep16

investigate database query time for blockFeatureLimits - limits of all blocks are requested when Pretzel frontend starts; this uses the cached value which takes ~300ms. Investigate the initial query time before the is cached; is it a limitation to scaling up the number of datasets / features.

2022Sep19

[x] [1H/0H] spreadsheet upload : don't remove "chr" from chromosome name This was done originally because commonly datasets had "chr" prefix, and Pretzel generally does not. But in some cases Pretzel dataset will have "chr" prefix, so we can stop removing the chr, and expect users to match chromosome names properly. resources/tools/dev/snps2Dataset.pl : if (! %chromosomeRenames) { $a[c_chr] =~ s/^chr//; ... (work-around tested OK : add a non-empty Chromosome Renaming worksheet; e.g. any text which will not match) This legacy feature was not carried over into JavaScript spreadsheet-read.js bda125fd.

2022Sep20

[ ] [2-3H] spreadsheet : spaces in file name not handled

2022Oct04 : spreadsheet upload : column names and gaps in error reporting to GUI :

[x] [1-2H] Alignment should be able to use Start column, not Position (the wiki says that an alignments needs Start, not Position.) spreadsheet with : Marker, Start, and Chromosome (check order) got error : Name column is required. Chromosome column is required. Position column is required. this worked : Name, Chromosome, Position usually an alignmment has Start, End Tested OK after conversion to JavaScript bda125fd.
- [ ] [1-2H] the error response is in HTML instead of JSON, which explains why it didn't get displayed in the GUI. check the API return type or callback.
[ ] [1-3H] QTL upload with Flanking Markers : currently need an Start / Position column, which can be empty
[ ] add a parentName check in the upload; currently it doesn't evaluate the parentName until loading the QTL in frontend

2022Oct06 :

[ ] [1-2H] numeric Flanking Markers e.g. 3270804 were not matched ('are not in parent'); work-around by formatting that cell to be "text" in MS Excel. Flanking Markers (and other feature name fields) should be read as text
[ ] [1-2H] quotes in column name ?
export 'columnsKeyString=name "Alternate SNP ID" "Unique SNP Locus ID" "SNP ' ... /app/lb3app/scripts/uploadSpreadsheet.bash: eval: line 186: unexpected EOF while looking for matching `"' ... think the column name had either a new line in it, or [ and ] were a problem
[ ] [1-2H] punctuation in column names; should be sanitized : ... "Total no. extendable hits possible for probe" "Target Chr" "Target bp" "Is SNP being assayed the target loci?" "Delete if reporting single locus?" ' ... insert_features_recursive key Total no. extendable hits possible for probe must not contain '.' undefined
- [ ] [1-2H] server exited after that error

2022Oct20

[x] [/20H] bda125fd Add parsing of uploaded spreadsheets within node server
- [x] [/3H] 1ea94549 : spreadsheet javascript : handle sheets with parentName column and hence which generate multiple datasets.
- [x] [/6H] b6c672e4 : spreadsheet javascript : completion of parentName column processing
- [x] [/4H] 1c7eea1e : after spreadsheet upload, display multiple dataset names, warnings, errors.
- [x] [/4H] d43b3914 : spreadsheet javascript : trim quotes, convert . to _, round position, etc
- [x] [/2H] 252c9ff6 : spreadsheet upload : filter out empty cells in metadata, apply rounding to other numeric columns
- [x] [/2H] 58ef76b8 : spreadsheet javascript : handle spaces around | in the sheet name, omit commented-out columns, handle empty sheet
- [x] [/2H] b3fa634c : spreadsheet : default namespace from parentName, meta.platform; remove . from metadata field names.
- [x] [/6H] 83104c85 : spreadsheet : add Alias worksheet.
- [x] [/1H] f530ed94 : spreadsheet upload : fix rounding - was creating string location values
- [x] [/1H] 62738af0 : spreadsheet upload : ensure that QTL check is done - parent contains named features
- [x] [/4H] b3385ecb : spreadsheet upload : show per-dataset warnings in GUI, sum the aliases delete count
- [x] [/2H] 169610e2 : spreadsheet upload : handle warnings when QTL parentName
- [x] [/3H] bd9d1ed5 : spreadsheet upload : filter out empty datasets; handle blank lines and comment-out lines in the same sheet

2022Nov25

[ ] [1-2H] error message was [object] [object], HTML < pre > in response
[ ] [1-2H] warning if [start, end] are not numbers, and skip the line (may be numbers within quotes, so strip spaces, quotes, etc first) test data : 221125_55k_marker_alignment_CSv1.0_Pretzel_Upload.xlsx

Don-Isdale commented 2 years ago

2022Oct24

[ ] [1H] 'V' Svg Export browser plugin shows the Pretzel SVG better than others - indicate this in wiki

in exported SVG :

[ ] [1-3H] axis ticks text centred on axis, maybe CSS not seen by export
[ ] [1-3H] switch off drop zones, could use CSS or maybe in publish mode, or transparent background
[ ] [1-2H] traits : colour only the selected / viewed traits; perhaps re-colour when change selection

Don-Isdale commented 2 years ago

small, useful features to add

from meeting 2022Oct24 :

[ ] [2-6H] zoom to an interval, i.e. paste in an interval
[ ] [2-6H] add a feature at a location, i.e. brush interval and create feature

from MS Teams, 2022Nov25-26 :

[ ] [2-4H] Features Search : support partial match, as an alternative to current (exact) match. e.g. : exact / partial / regexp (radio buttons), and a toggle button for case sensitive
[ ] [1-2H] add text "All Groups" to the group dropdown menu as the default rather than blank.

Don-Isdale commented 2 years ago

2022Oct27

[x] [/5H] ensured that QTL features are loaded when axis is added. 1bc4a38c

2022Dec15

[ ] 'New Datasource' dialog : [1H] show a message 'Connecting ...' or 'Failed to Connect' (maybe : [2-3H] make 'Connect' button insensitive after clicking, until fail or close.)

Don-Isdale commented 1 year ago

2023Jan05

[4-8H] feature table : sort by Position : should leave the Axes separate, i.e. sort by Axis then Position Multiple Blocks within 1 axis will be merged together when sorting by Position. (possibly a custom sort function on Position column, which uses Block + Position as key). Sort by Feature Name can merge the Blocks.

Don-Isdale commented 1 year ago

2023Jan13

[x] [/3H] handle disparate axes length in filter pathIsSyntenic(). dd6da05c This issue arose before v2.15.2, likely when adding : distance to neighbours using both axes, in 27e9d4c0

2023Jan17

[ ] [2-4H] In Explorer : 'All Datasets' there is not sufficient bottom margin to access the Delete button at the bottom of the Dataset right-click menu.

MicrosoftTeams-image (4) work-arounds :

click + on the dataset, which displays the dataset blocks and creates vertical space to see the other menu items
use down-arrow and count (approx 3 steps down to the delete button)
[x] [2-3H/2H] spreadsheet upload : report to the user any work-sheets for which metadata was not found and vice versa, because it is easy to put '_' in one name and ' ' in the other and not notice why they don't match 3bc05106

[ ] Currently when a user tries to sign up again, Pretzel frontend does not show any message. Given that confirmation emails are getting lost in junk folders, it might be good to : [2-8H] resend the confirmation email at that time [2-3H] and show a message on the GUI.
- [ ] related : test whether it is possible to request a reset password without having clicked on the address confirmation email. If so, perhaps remind the user at that time and maybe resend the email.

2023Feb03

[ ] [2-4H] (paths colinearity) check that alias paths are handled in components/draw/block-adj.js : pathIsSyntenic()

Don-Isdale commented 1 year ago

2023Feb08 : v2.16.0 on main and pulses

[ ] [2-6H] featuresCounts bins not displayed after brush & click Zoom, then click Reset. data : v1.0_DArT_gbs_markers. [08 Feb 2023 11:19, Teams channel : Pretzel Chats] This is with 90k markers also viewed.I checked in the DOM, and it is not just shifted left (onto the left track, which we have seen), there are no bins displayed

Don-Isdale commented 1 year ago

2023May24 (wheat proteome meeting) :

[x] [2-6H/2H] user login : add toggle to enable the password to be visible while typing it 43b88e2b user-login : add a checkbox to Show Password
[x] investigate if possible to increase server log to indicate the cause of login issue in loopback auth. 25c24862 [/2H] user-login : handle change in API result code, and update arg display. a70f7e41 [/4H] user authentication : handle receiving an HTML error page instead of jSON a218e17b [/0.5H] user authentication : show error from HTML error page in GUI
[ ] [1-3H] QTLs not initially displayed (F020443 on dirac2, on HC genes)
[ ] [1-3H] while zooming some QTL rectangles remain displayed when they should have exited
investigate if possible to change email login authentication (loopback) to match email regardless of case, i.e. case insensitive

Don-Isdale commented 10 months ago

2023Dec08

4e10cfcb handle undefined server, seen when sending sign-up f601b748 re-enable graph listening to window resize event 4f38c159 handle different name and scope in blast search

plantinformatics / pretzel

Fixes/tweaks to existing features #216

Comments with open items :construction:

small, useful features to add

use down-arrow and count (approx 3 steps down to the delete button)