ifpd_serve: "ERROR 500: nothing here, sorry :("

[christianddietrich]

Describe the bug After a database has been added, ifish (ifpd_serve) only responds with "ERROR 500: nothing here, sorry :(". If the database folder is empty everything works fine.

To Reproduce

1. ifpd_mkdb -O /data/ifish/static/db/hg19_chr1 /data/ifish/iFISH.40mer.hg19.chr1 hg19 hg19_chr1
2. ifpd_serve -u 127.0.0.1 -s "/data/ifish/static"
3. Open "http://127.0.0.1:8080/"

Expected behavior HTTP 200 / webinterface works

Desktop (please complete the following information):

• OS: Debian 9.9 x86_64 / Python 3.7.4 / iFISH 2.0.3
• Browser: Chrome 75 / Firefox 68

Additional context

(ifishvenv) ifish@localhost:~$ ifpd_mkdb -O /data/ifish/static/db/hg19_chr1 /data/ifish/iFISH.40mer.hg19.chr1 hg19 hg19_chr1
2019-07-24 13:32:28,404   root INFO     Buffering bed-like file...
2019-07-24 13:32:28,736   root INFO     Writing database...
2161644it [01:52, 19133.60it/s]                                                                                                                                                                                                                                                                                                                                            
2019-07-24 13:34:21,714   root INFO     Sorting oligos...
2019-07-24 13:34:41,331   root INFO     Generating config file...
2019-07-24 13:34:41,332   root INFO     Created database "hg19_chr1".

(ifishvenv) ifish@localhost:~$ cat static/db/hg19_chr1/.config 
[DATABASE]
name = hg19_chr1
refgenome = hg19

[OLIGOS]
sequence = False
min_dist = 11
min_length = 39
max_length = 39
overlaps = False

[SOURCE]
bed = /data/ifish/iFISH.40mer.hg19.chr1
enforced2bed3 = False
outdirectory = /data/ifish/static/db/hg19_chr1

(ifishvenv) ifish@localhost:~$ ifpd_serve -u 127.0.0.1 -s "/data/ifish/static"
Bottle v0.12.17 server starting up (using PasteServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

(worker 0 ) Started new worker 139754498242304: Initial worker pool
(worker 3 ) Started new worker 139754026358528: Initial worker pool
(worker 2 ) Started new worker 139754034751232: Initial worker pool
(worker 1 ) Started new worker 139754506635008: Initial worker pool
(worker 4 ) Started new worker 139754017965824: Initial worker pool
(worker 5 ) Started new worker 139753672406784: Initial worker pool
(worker 6 ) Started new worker 139753664014080: Initial worker pool
(worker 7 ) Started new worker 139753655621376: Initial worker pool
(worker 8 ) Started new worker 139753647228672: Initial worker pool
serving on http://127.0.0.1:8080
(worker 9 ) Started new worker 139753638835968: Initial worker pool
(MainThread) Added task (0 tasks queued)
/data/ifish/ifishvenv/bin/bottle.py:3224: DeprecationWarning: Absolute template path names are deprecated.
  self.filename = self.search(self.name, self.lookup)
Traceback (most recent call last):
  File "/data/ifish/ifishvenv/bin/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 1742, in wrapper
    rv = callback(*a, **ka)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3648, in wrapper
    return template(tpl_name, **tplvars)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3624, in template
    return TEMPLATES[tplid].render(kwargs)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3413, in render
    self.execute(stdout, env)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3400, in execute
    eval(self.co, env)
  File "/data/ifish/ifishvenv/lib/python3.7/site-packages/ifpd/sections/probe_design/views/home.tpl", line 77, in <module>
    % include(vpath + 'databases.tpl')
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3390, in _include
    return self.cache[_name].execute(env['_stdout'], env)
  File "/data/ifish/ifishvenv/bin/bottle.py", line 3400, in execute
    eval(self.co, env)
  File "/data/ifish/ifishvenv/lib/python3.7/site-packages/ifpd/sections/probe_design/views/databases.tpl", line 36, in <module>
    %for k in [k for k in config['CUSTOM'].keys() if not k in ['reference', 'url']]:
  File "/data/ifish/python/lib/python3.7/configparser.py", line 958, in __getitem__
    raise KeyError(key)
KeyError: 'CUSTOM'
127.0.0.1 - - [24/Jul/2019:13:44:06 +0200] "GET /probe-design/ HTTP/1.1" 500 33 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"

Thanks for your help! It may be that I am just doing something wrong, I have tried to stick to the documentation.

[ggirelli]

Hi there, thanks for the report and sorry for the inconvenience. It looks like a mistake from my side. Could you share the .config file that should have been generated in the database folder? That would give me the details I need to fix this :)

[christianddietrich]

Hi there, thanks for the report and sorry for the inconvenience. It looks like a mistake from my side. Could you share the .config file that should have been generated in the database folder? That would give me the details I need to fix this :)

That was quick.

.config:

[DATABASE]
name = hg19_chr1
refgenome = hg19

[OLIGOS]
sequence = False
min_dist = 11
min_length = 39
max_length = 39
overlaps = False

[SOURCE]
bed = /data/ifish/iFISH.40mer.hg19.chr1
enforced2bed3 = False
outdirectory = /data/ifish/static/db/hg19_chr1

.log:

2019-07-24 13:32:28,404   root INFO     Buffering bed-like file...
2019-07-24 13:32:28,736   root INFO     Writing database...
2019-07-24 13:34:21,714   root INFO     Sorting oligos...
2019-07-24 13:34:41,331   root INFO     Generating config file...
2019-07-24 13:34:41,332   root INFO     Created database "hg19_chr1".

[ggirelli]

Hi, thanks for the .config file! I have released the new v2.0.3.post1 version that should patch this up. Depending on how you installed the package, you can either pull from the repo or install from pypi with pip. You might also want to run pip with the --no-cache-dir option to avoid issues with cached package indexes. Let me know if it works now and I will close this ticket. :)

[christianddietrich]

Hi, thanks for the .config file! I have released the new v2.0.3.post1 version that should patch this up. Depending on how you installed the package, you can either pull from the repo or install from pypi with pip. You might also want to run pip with the --no-cache-dir option to avoid issues with cached package indexes. Let me know if it works now and I will close this ticket. :)

Hi,

With the new version it works fine, thanks!

Maybe you can tell me how can I add multiple chromosomes to a database? (As you do it on http://www.ifish4u.org/probe-design/)

(ifishvenv) ifish@localhost:~$ ifpd_mkdb -O /data/ifish/static/db/hg19_chr1 /data/ifish/iFISH.40mer.hg19.chr2 hg19 hg19_chr1
Traceback (most recent call last):
  File "/data/ifish/ifishvenv/bin/ifpd_mkdb", line 153, in <module>
    assert not os.path.isdir(args.O), assert_msg
AssertionError: output folder already exists: "/data/ifish/static/db/hg19_chr1"

Thanks!

[ggirelli]

Glad to hear it works now, sorry again for the inconvenience.

To generate a database with multiple chromosomes you should give a bed file with oligos from multiple chromosomes as input to the ifpd_mkdb script. The script then takes care of splitting the inpu accordingly to the ifpd format.

Basically, if you go here and download the hg19 file, you will see that that bed file contains multiple chromosomes. You can then rebuild the database locally from that file.

If using all chromosomes is overkill or, for any reason, you prefer to not do it and you would like to have only a few, you can always do one of the following:

1. Download the hg19 full database from the aforementioned link, and then remove all rows that do not match your chromosomes of interest from the bed file.
2. Download only the chromosomes you are interested in and merge the bed files with cat chr1 chr2 chrX > merged.bed or similar.

And then run ifpd_mkdb.

Hope this helps. I'll close the thread but feel free to comment here if in need of help on this topic. :)