Dataset location in the Hadoop including DS name

[acanakoglu]

In the repository management, we are adding to the repository with the full path of the files, without mentioning the DS name.

This would create problems two datasets, which are loading from the local folder. They are sharing the same folder in the Hadoop. This error occurred while using the GMQL-Importer package.

The proposed solution which is approved by @akaitoua, is as below: The dataset will be saved by adding the dataset name in the path as in the example: /user/{username}/regions/{DatasetName}/*.bed(.meta)

[akaitoua]

Actually this is the way that the new datasets are generated in the repository directory.

[acanakoglu]

Hi @akaitoua , The correction has some bugs. Could you please check it? You can check with uploading any dataset to the system.

java.lang.NullPointerException: null
        at java.io.File.<init>(File.java:277)
        at it.polimi.genomics.repository.FSRepository.DFSRepository$$anonfun$importDs$1.apply(DFSRepository.scala:71)
        at it.polimi.genomics.repository.FSRepository.DFSRepository$$anonfun$importDs$1.apply(DFSRepository.scala:70)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at it.polimi.genomics.repository.FSRepository.DFSRepository.importDs(DFSRepository.scala:70)
        at controllers.gmql.DSManager.controllers$gmql$DSManager$$importDataset(DSManager.scala:873)
        at controllers.gmql.DSManager$$anonfun$uploadSample$1.apply(DSManager.scala:770)
        at controllers.gmql.DSManager$$anonfun$uploadSample$1.apply(DSManager.scala:741)
        at play.api.mvc.ActionBuilder$$anonfun$apply$16.apply(Action.scala:408)
        at play.api.mvc.ActionBuilder$$anonfun$apply$16.apply(Action.scala:407)
        at wrappers.authanticate.AuthenticatedAction$.invokeBlock(AuthenticatedAction.scala:33)
        at wrappers.authanticate.AuthenticatedAction$.invokeBlock(AuthenticatedAction.scala:26)
        at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:493)
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105)
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105)
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:104)
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:103)
        at scala.Option.map(Option.scala:146)
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:103)
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:96)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:537)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:537)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[akaitoua]

@acanakoglu, i am still not able to test it but i tried to fix it.

[acanakoglu]

Thank you @akaitoua. I tested and it is working. I will test with @nachodox importer implementation as soon as I can.