Merge calling card functionality

[arnavm]

Hi guys,

Here is the pull request to introduce calling card functionality into the Epigenome Browser. I've tested this locally and seems to perform without any problems. After the merge, we should meet regularly to touch base on developing this track type further. Let me know if you have any questions.

-- Arnav

[smhsu]

I have two questions:

1. What's with browser/js/base.js.bak? git already stores backups/multiple versions.
2. Would it be possible to move your test files to test/callingCard/?

[arnavm]

1. That was a backup of my original base.js file. The easiest way to bring my repo up to date was to modify the latest base.js, but I needed to store my old changes somewhere. This file can now be deleted.

2. Yes, that’s fine. The test files don’t get called by the scripts, they’re just there for validation in case something breaks.

-- Arnav

On Feb 13, 2018, 11:32 AM -0600, Silas Hsu notifications@github.com, wrote:

I have two questions:

1. What's with browser/js/base.js.bak? git already stores backups/multiple versions.
2. Would it be possible to move your test files to test/callingCard/?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[arnavm]

Bump. Checking in on this.

[smhsu]

Sorry about that! I came to work this morning realizing I forgot to work on this yesterday. Testing locally and on the test server today (epgg-test.wustl.edu/browser).

[smhsu]

Update: something in your code is breaking database connections. I'm working on it but don't know how long it will take to fix, or if there will be other issues after that.

[arnavm]

Interesting. I'm Docker the MySQL connections worked fine. I can't remember if I had to tweak something to get it to work though. What's the error message?

-- Arnav

On Feb 20, 2018, 1:19 PM -0600, Silas Hsu notifications@github.com, wrote:

Update: something in your code is breaking database connections. I'm working on it but don't know how long it will take to fix, or if there will be other issues after that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[smhsu]

cannot connect to db (hg19)

I know, not very useful

[smhsu]

I've fixed the database issue on http://epgg-test.wustl.edu/browser/ by reverting commit 4e4a5c181bb1ef3a807e476c057def7a1d373b60. Umm... I don't know how to actually add a calling card track; could you go to the test server and try it out?

[arnavm]

Ahaha, I do remember changing the db username, but now I can't remember why I had to do that. Probably didn't want to clobber any existing data.

I've tried uploading a working JSON on the test browser but it's giving me an error: Unknown type in datahub: callingcard

I've tried refreshing the cache but that hasn't fixed the issue. Does the test browser code reflect the update?

[smhsu]

Sorry about that; forgot to copy a .js file. Try it now?

[arnavm]

That seems to have done the trick. Calling card files are being properly loaded and visualized. Gene search and gene set views are good. Pan and zoom are behaving well.

Unfortunately, SVG output isn't working. I suspect I'm incorrectly specifying the coordinates for the data points. I'll keep working on this. In the meantime, should I close out this pull request and send another one when SVG is working, or should I keep this open for the time being?

[smhsu]

Let's keep it open for now. There's still a few things to fix besides SVG, such as:

• Your code doesn't compile without the std=c99 flag
• Your code still triggers a bunch of compiler warnings

[smhsu]

Btw, if you give me your json, I can try my hand at debugging.

[arnavm]

Sure, I'd love to get another set of eyes on the code. Here's a sample JSON (1 calling card track, 2 bedgraphs). Apparently Github only lets you attach text files if they end in .txt.

Sample.json.txt

[arnavm]

I should add that the genome is hg38 for that JSON.

[arnavm]

I'm guessing the problem is in lines 11249-11256 of base.js. Seems to me that the code should be:

svgdata.push({ type: svgt_arc, radius: size, x1: position, y1: bary, x2: position + size, y2: 0, color: barcolor })

Though it's not apparent to me what x2 and y2 specify.

[arnavm]

Also, can you send me the Apache conf file for the test server? I can try debugging SVG with my local Docker version, but for some reason SVG generation doesn't work. It throws an ERR_EMPTY_RESPONSE error when you try. Seems like Apache is mishandling the POST request, or can't execute postdeposit on the server. I want to rule out whether it's something about how Apache is configured.

[arnavm]

SVG output has been fixed in commit 0c0f5220ae671c0e9b7c2378a59738bbfb37c3bb. The svgt_circle case hadn't been defined in base.js:Browser.prototype.svgadd(), so I added that functionality. It doesn't do the kind of bounds checking that the other types do, but this was the only way I could get it to work. So now, calling card tracks successfully generate SVG files. It can't make PDFs, but I suspect that's a Docker-specific thing, probably a missing PS/PDF library. I'll look into it but it's low priority.

I also figured out why the Docker version wasn't allowing us to generate SVGs in the first place. The path specified in postdeposit.c differs between the deployment and Docker versions. It originally only pointed to the deployment path, which is not the path the Docker version uses. I added the Docker line in 953a48a905006145f84dda748fedb6e61c33b4ec. Depending on whether you're using the deployment or Docker versions, you can uncomment the appropriate line.

Let me know when you've merged these into the test server and I'll confirm whether everything works.

[smhsu]

Installed on test server. Please note that in fileload.js there is some code floating outside any function which is causing an error on line 992.

[arnavm]

Fixed PDF generation. Same problem as before: WWWT path in subtleKnife.c was set to deployment version, not Docker version. Added line for Docker version; comment/uncomment as needed.

[smhsu]

Great! We're almost there. You just need to fix compiler warnings (not all of them, just the new ones :p). make clean followed by make to see them.

For example: subtleKnife.c: In function ‘callingCardSort_startAsc’: subtleKnife.c:698:6: warning: assignment from incompatible pointer type [enabled by default]

[arnavm]

Sounds do-able. I'll try to have that updated by the weekend. In the meantime, everything seems to be working fine on the test server, including SVG and PDF generation.

[arnavm]

Last new feature, for the time being. Calling card files can now be directly uploaded using the file uploader and visualized on the browser. Marked as experimental but no obvious issues came up while I was testing.

Next task: fix compiler warnings.

[arnavm]

I think I've taken care of all new compiler warnings. Take a look and let me know if you find something I've overlooked.

[smhsu]

Thanks for all the hard work! I will be away on Thursday and Friday, so the next time I will be able to take a look will be on Monday. If you need something urgently, please ping Daofeng.

[smhsu]

Looks good to me! Daofeng says he'll do some final testing tonight, and once he gives the thumbs-up, we'll do the merge.

All your most recent changes are on the test server right now, so you can play with it there if you want to as well.

[arnavm]

I've been playing around with the deployment version of the browser at http://epigenomegateway.wustl.edu/browser/ and I can't get calling card tracks to load. It gives me the following error: "hg38 Missing track for reordering." The same JSON file works on both the test browser and the Docker version but doesn't work on the official site. Any ideas what could be going on? Attached a working JSON. test_json.txt

[lidaof]

I can load your test datahub file on the public site. can you retry please? thanks.

[arnavm]

Hm, yes, now it works. Not sure what changed, but looks like everything is fine. I'll let you know if I run across any other problems.