singularity -- unable to open database file in shinyapps

[nschiraldi]

I am trying to run idep locally via the singularity build. When I open any of the shiny apps, they load but then disconnect. Any help is greatly appreciated!

For example, if I run any of the idep shiny apps (e.g. host:port/idep90/), in the logs I am seeing:

Warning: Error in connection_connect: Could not connect to database:
unable to open database file
  58: <Anonymous>
Error in connection_connect(dbname, loadable.extensions, flags, vfs) :
  Could not connect to database:
unable to open database file

These are the files in /srv/shiny-server/idep90:

Singularity> ls /srv/shiny-server/idep90
fb.js  ga.js  iDEP_core_functions.R  readme.md  server.R  ui.R  www

And /srv/data/data92 has been bound correctly.

Singularity> ls /srv/data/data92
convertIDs.db  data_go  geneInfo  motif  pathwayDB

I edited the idep-server startup to listen on all interfaces:

server {
  listen $port 0.0.0.0;
  location / {
    site_dir /srv/shiny-server;
    directory_index on;
    app_idle_timeout 900;
    log_dir /var/log/shiny-server;
  }
}

I modified the singularity-standalone/buid_idep.sh file so that it could be built without root permissions. I built the idep.def file in singularity hub. If you use this build script and execute to build in /path/to/idep/ you can recreate the environment I am working from:

#! /bin/bash

# Build script to set up a single instance of the idep shiny server running in
# a singularity container. No kind of load balancing is done - only the
# built-in shiny web server is running **as the user**

################################################################################
#                                  Functions                                   #
################################################################################
function fail() {
    printf "### ERROR ### %s\n" "$@" >&2
    exit 1
}

function info() {
    printf "### INFO  ### %s\n" "$@" >&2
}

function download() {
    local name="${1}"
    local url="${2}"
    if [[ -e "${name}" ]]; then
        info "${name} already exists - skipping download"
    else
        info "downloading ${name}"
        tmp="${name}.tgz"
        (
          wget -O "${tmp}" "${url}" \
          && tar -xzf "${tmp}" \
          && rm -f "${tmp}" \
        ) || fail "download failed"
    fi
}

################################################################################
#                                   Settings                                   #
################################################################################
wd="$(pwd)"
def_file="${wd}/idep.def"
img_file="${wd}/idep.sif"
wrapper="${wd}/idep-server"

################################################################################
#                                     Main                                     #
################################################################################
prefix="${1:-none}"

if [[ "${prefix}" == "none" ]]; then
    printf "USAGE: build_idep <install prefix>\n"
    exit 1
fi

###
### set up the folder structure for the application
###
cd "${prefix}" || fail "prefix for install does not exist"
mkdir -p libexec share/idep/data bin || fail "could not create directories in ${prefix}"

cp "${wrapper}" bin

###
### build the container - this takes a long time
###

if command -v singularity &> /dev/null; then
    info "Found singularity version $(singularity --version)"
else
    fail "singularity not found"
fi

if [[ ! -f libexec/idep.sif ]]; then
    if [[ -f "${img_file}" ]]; then
        info "using existing singularity image in build script directory"
        cp "${img_file}" libexec/idep.sif
    else
        [[ -f "${def_file}" ]] || fail "container definition file does not exist"
        info "Building container"
        #sudo singularity build libexec/idep.sif share/idep/idep.def || fail "singularity build failed"
        singularity pull libexec/idep.sif shub://nschiraldi/singularity:idep || fail "singularity pull failed"
    fi
    cp "${def_file}" share/idep
else
    info "Container already exists"
fi

###
### fetch the data
###
cd share/idep/data || fail "could not cd to ${prefix}/share/idep/data"
download pathwayDB https://sdsu.box.com/shared/static/c24f792ojoikpzu0lkpng8uuf9ychwm7.gz
download motif https://sdsu.box.com/shared/static/9v1ao6mwhduvrcx793j3answph9gqnkt.gz
download geneInfo https://sdsu.box.com/shared/static/mns0k1uvwtfnsohoc89b984ih36nmnz9.gz
download data_go https://sdsu.box.com/shared/static/qwpdh36vcisgy1hcmadck8i8ezhvr2fh.gz
download convertIDs.db https://sdsu.box.com/shared/static/sorewt7w6iypmhg2k2xhyi8myeit156o.gz

[nschiraldi]

Found a few issues, putting here for future users. It looks like the data downloads listed in the build_idep.sh do not have all of the files.

First, singularity should be run with:

singularity exec \
    -B "${tmp}/shiny-server.conf:/etc/shiny-server/shiny-server.conf" \
    -B "${tmp}/varlog:/var/log/shiny-server" \
    -B "${tmp}/varlib:/var/lib/shiny-server" \
    -B "${datadir}:/srv/data/data96" \
    -B "${tmp}:/tmp" \
    ${image} shiny-server

Second, the downloads from box don't seem to pull down this file: '../../data/data96/data_go/STRING11_species.csv'

Would it be possible to get links to the updated box data download?

[HalforcNull]

Hi @wresch, Could you help us analysis this issue?

[wresch]

I'll have a look. Catching up after vacation so it may take me a bit

On Tue, Aug 27, 2019 at 2:27 PM Halforc Null notifications@github.com wrote:

Hi @wresch https://github.com/wresch, Could you help us analysis this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iDEP-SDSU/idep/issues/74?email_source=notifications&email_token=AACCS6GWDLE4AGN7HUL2ESLQGVWYLA5CNFSM4HZTBKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5IV2GY#issuecomment-525425947, or mute the thread https://github.com/notifications/unsubscribe-auth/AACCS6H4MBE57U6AW2CDTZTQGVWYLANCNFSM4HZTBKUQ .

[nschiraldi]

Thanks, please let me know if you think I'm doing something wrong! I'll need to refresh myself on this as well, but I think the biggest issue was the file pull from box.