search

G-Node / gin-proc

BSD 3-Clause "New" or "Revised" License
0 stars 7 forks source link

Basic Prototype [Draft] #14

Closed mrinalwahal closed 5 years ago

mrinalwahal commented 5 years ago

Added manual file inputs. However, I'm getting merge errors while pushing to gin-proc branch.

achilleas-k commented 5 years ago

I'm not sure I understand how merge errors (push reject) are happening. Starting from a fresh clone (which the container starts in after the Clone step) and going through the steps:

Where's the conflict coming from? <branch> is freshly cloned, a commit was added, so a push should be clean.

achilleas-k commented 5 years ago

Just started reading through the PR but a general comment:

Please don't check in Python cache (__pycache__) and Python environments (env) to the repository.

achilleas-k commented 5 years ago

One more issue we just discovered:

The Python gogs_client seems to be very much out of date. The latest release was June 2017 and that might be an issue going forward. Just something to keep in mind.

achilleas-k commented 5 years ago

SSH key setup fixes described here: https://github.com/G-Node/gin-proc/issues/12#issuecomment-514716892

mrinalwahal commented 5 years ago

All that is done: [1] Most of the OS commands have been changed to subprocess. Except one. [2] Added filenames in inverted commas to handle filenames with spaces between them. (Though this throws errors in certain cases like file paths. But works fine during git push) [3] Repo is cleaned and is now divided between the front-end and back-end code. pycache, env, et cetera is removed. [4] Drone config works. Pushes the output files the user chooses in GUI to a new folder by name of build-job in gin-proc branch. [5] You can now choose which files to annex from front-end. [6] GiN's address isn't harcoded anymore. You have to export the IP before starting flask [7] Drone config now copies the output files to a temporary location and copies them back during the push. [8] You can now enter multiple files from the front-end.

All that is left: [1] Using the new keys doesn't work yet. GIT_SSH_COMMAND hasn't worked for me yet. So you'll have to set up your keys with GIN before testing. I've skipped this part in the code. [2] In certain cases the Drone config is not written properly. Whilst in certain cases it is. I'm trying to figure this out. [3] Caching of building (updating and restoring both) may not work properly, this is because Snakemake produces its intermediate files within the project folder itself, and I'm finding a way for it to change its location outside the project folder -- to our actual caching location. [4] Micro-service isn't yet dockerized, so you'll have to set-up the thing manually just like last time. My dockerfile isn't working properly, and I didn't want to delay the push just because of it.

Here are the updated documented steps for starting gin-proc.

mrinalwahal commented 5 years ago

One more issue we just discovered:

The Python gogs_client seems to be very much out of date. The latest release was June 2017 and that might be an issue going forward. Just something to keep in mind.

I'll remove this python package as soon as I'm able to log in to GOGS API directly from its endpoint. Waiting for your help on that end. I can't find the API endpoint.