search

gvegayon / parallel

PARALLEL: Stata module for parallel computing
https://rawgit.com/gvegayon/parallel/master/ado/parallel.html
MIT License
117 stars 26 forks source link

Connection issues on OSX with Stata MP 13.1 #50

Closed gvegayon closed 7 years ago

gvegayon commented 7 years ago

Expected behavior and actual behavior

"[C]annot even run the examples you provide in your help file. The clusters never report back; the program just keeps running forever and I have to hit Break to stop it. I have of course specified the number of cores before I run anything." (Reported by Belinda Foster)

Steps to reproduce the problem

Examples

System information

dual core Mac running Stata MP 13.1

Output from creturn list:

System values
    -------------------------------------------------------------------------------------------------------
       c(stata_version) = 13.1
             c(version) = 13.1                       (version)
    -------------------------------------------------------------------------------------------------------
           c(born_date) = "09 Dec 2015"
              c(flavor) = "IC"
                 c(bit) = 64
                  c(SE) = 1
                  c(MP) = 1
          c(processors) = 2                          (Stata/MP, set processors)
      c(processors_lic) = 8
     c(processors_mach) = 2
      c(processors_max) = 2
                c(mode) = ""
             c(console) = ""
    -------------------------------------------------------------------------------------------------------
                  c(os) = "MacOSX"
               c(osdtl) = "10.6.8"
            c(hostname) = "iMac.local"
        c(machine_type) = "Macintosh (Intel .."
           c(byteorder) = "lohi"

    -------------------------------------------------------------------------------------------------------

Directories and paths

    -------------------------------------------------------------------------------------------------------
        c(sysdir_stata) = "/Applications/Sta.."      (sysdir)
         c(sysdir_base) = "/Applications/Sta.."      (sysdir)
         c(sysdir_site) = "/Applications/Sta.."      (sysdir)
         c(sysdir_plus) = "~/Library/Applica.."      (sysdir)
     c(sysdir_personal) = "~/Library/Applica.."      (sysdir)
     c(sysdir_oldplace) = "~/ado/"                   (sysdir)
              c(tmpdir) = "/var/folders/4j/4.."
    -------------------------------------------------------------------------------------------------------
             c(adopath) = "BASE;SITE;.;PERSO.."      (adopath)
              c(dirsep) = "/"
    -------------------------------------------------------------------------------------------------------

System limits

    -------------------------------------------------------------------------------------------------------
        c(max_N_theory) = 2147483647
        c(max_k_theory) = 5000                       (set maxvar)
    c(max_width_theory) = 1048576                    (set maxvar)
    -------------------------------------------------------------------------------------------------------
         c(max_matsize) = 11000
         c(min_matsize) = 10
    -------------------------------------------------------------------------------------------------------
        c(max_macrolen) = 1081511
            c(macrolen) = 165200                     (set maxvar)
          c(max_cmdlen) = 1081527
              c(cmdlen) = 165216                     (set maxvar)
             c(namelen) = 32
               c(eqlen) = 399
    -------------------------------------------------------------------------------------------------------

Numerical and string limits

    -------------------------------------------------------------------------------------------------------
           c(mindouble) = -8.9884656743e+307
           c(maxdouble) = 8.9884656743e+307
           c(epsdouble) = 2.22044604925e-16
      c(smallestdouble) = 2.2250738585e-308
    -------------------------------------------------------------------------------------------------------
            c(minfloat) = -1.70141173319e+38
            c(maxfloat) = 1.70141173319e+38
            c(epsfloat) = 1.19209289551e-07
    -------------------------------------------------------------------------------------------------------
             c(minlong) = -2147483647
             c(maxlong) = 2147483620
    -------------------------------------------------------------------------------------------------------
              c(minint) = -32767
              c(maxint) = 32740
    -------------------------------------------------------------------------------------------------------
             c(minbyte) = -127
             c(maxbyte) = 100
    -------------------------------------------------------------------------------------------------------
        c(maxstrvarlen) = 2045
       c(maxstrlvarlen) = 2000000000
        c(maxvlabellen) = 32000
    -------------------------------------------------------------------------------------------------------

Current dataset

    -------------------------------------------------------------------------------------------------------
                   c(N) = 74
                   c(k) = 12
               c(width) = 43
             c(changed) = 0
            c(filename) = "/Applications/Sta.."
            c(filedate) = "13 Apr 2013 17:45"
    -------------------------------------------------------------------------------------------------------

Memory settings

    -------------------------------------------------------------------------------------------------------
              c(memory) = 33554432
              c(maxvar) = 5000                       (set maxvar)
             c(matsize) = 400                        (set matsize)
            c(niceness) = 5                          (set min_memory)
          c(min_memory) = 0                          (set min_memory)
          c(max_memory) = .                          (set max_memory)
         c(segmentsize) = 33554432                   (set segmentsize)
    -------------------------------------------------------------------------------------------------------

Output settings

    -------------------------------------------------------------------------------------------------------
                c(more) = "off"                      (set more)
                c(rmsg) = "off"                      (set rmsg)
                  c(dp) = "period"                   (set dp)
            c(linesize) = 107                        (set linesize)
            c(pagesize) = 18                         (set pagesize)
             c(logtype) = "smcl"                     (set logtype)
             c(noisily) = 1
             c(charset) = "mac"                      (set charset)
             c(eolchar) = "unix"                     (set eolchar)
          c(notifyuser) = "on"                       (set notifyuser)
             c(playsnd) = "off"                      (set playsnd)
      c(include_bitmap) = "on"                       (set include_bitmap)
    -------------------------------------------------------------------------------------------------------
               c(level) = 95                         (set level)
    -------------------------------------------------------------------------------------------------------
      c(showbaselevels) = ""                         (set showbaselevels)
      c(showemptycells) = ""                         (set showemptycells)
         c(showomitted) = ""                         (set showomitted)
             c(fvlabel) = "on"                       (set fvlabel)
              c(fvwrap) = 1                          (set fvwrap)
            c(fvwrapon) = "word"                     (set fvwrapon)
            c(lstretch) = ""                         (set lstretch)
    -------------------------------------------------------------------------------------------------------
             c(cformat) = ""                         (set cformat)
             c(sformat) = ""                         (set sformat)
             c(pformat) = ""                         (set pformat)
    -------------------------------------------------------------------------------------------------------
      c(coeftabresults) = "on"                       (set coeftabresults)

Interface settings

    -------------------------------------------------------------------------------------------------------
          c(reventries) = 5000                       (set reventries)
         c(revkeyboard) = "on"                       (set revkeyboard)
         c(varkeyboard) = "on"                       (set varkeyboard)
         c(smoothfonts) = "on"                       (set smoothfonts)
             c(linegap) = 1                          (set linegap)
       c(scrollbufsize) = 204800                     (set scrollbufsize)
               c(maxdb) = 50                         (set maxdb)
    -------------------------------------------------------------------------------------------------------

Graphics settings

    -------------------------------------------------------------------------------------------------------
            c(graphics) = "on"                       (set graphics)
              c(scheme) = "s2color"                   (set scheme)
          c(printcolor) = "automatic"                (set printcolor)
           c(copycolor) = "automatic"                (set copycolor)
    -------------------------------------------------------------------------------------------------------

Efficiency settings

    -------------------------------------------------------------------------------------------------------
             c(adosize) = 1000                       (set adosize)
    -------------------------------------------------------------------------------------------------------

Network settings

    -------------------------------------------------------------------------------------------------------
            c(checksum) = "off"                      (set checksum)
            c(timeout1) = 30                         (set timeout1)
            c(timeout2) = 180                        (set timeout2)
    -------------------------------------------------------------------------------------------------------
           c(httpproxy) = "off"                      (set httpproxy)
       c(httpproxyhost) = ""                         (set httpproxyhost)
       c(httpproxyport) = 80                         (set httpproxyport)
    -------------------------------------------------------------------------------------------------------
       c(httpproxyauth) = "off"                      (set httpproxyauth)
       c(httpproxyuser) = ""                         (set httpproxyuser)
         c(httpproxypw) = ""                         (set httpproxypw)
    -------------------------------------------------------------------------------------------------------

Update settings

    -------------------------------------------------------------------------------------------------------
        c(update_query) = "off"                      (set update_query)
     c(update_interval) = 7                          (set update_interval)
       c(update_prompt) = "off"                      (set update_prompt)
    -------------------------------------------------------------------------------------------------------

Trace (program debugging) settings

    -------------------------------------------------------------------------------------------------------
               c(trace) = "off"                      (set trace)
          c(tracedepth) = 32000                      (set tracedepth)
            c(tracesep) = "on"                       (set tracesep)
         c(traceindent) = "on"                       (set traceindent)
         c(traceexpand) = "on"                       (set traceexpand)
         c(tracenumber) = "off"                      (set tracenumber)
         c(tracehilite) = ""                         (set tracehilite)
    -------------------------------------------------------------------------------------------------------

Mata settings

    -------------------------------------------------------------------------------------------------------
          c(matastrict) = "off"                      (set matastrict)
            c(matalnum) = "off"                      (set matalnum)
        c(mataoptimize) = "on"                       (set mataoptimize)
           c(matafavor) = "space"                    (set matafavor)
           c(matacache) = 400                        (set matacache)
            c(matalibs) = "lmatabase;lmataad.."      (set matalibs)
         c(matamofirst) = "off"                      (set matamofirst)
    -------------------------------------------------------------------------------------------------------

Other settings

    -------------------------------------------------------------------------------------------------------
                c(type) = "float"                    (set type)
             c(maxiter) = 16000                      (set maxiter)
       c(searchdefault) = "all"                      (set searchdefault)
                c(seed) = "X075bcd151f1.."      (set seed)
         c(version_rng) = 12.1
           c(varabbrev) = "on"                       (set varabbrev)
          c(emptycells) = "keep"                     (set emptycells)
             c(odbcmgr) = "iodbc"                    (set odbcmgr)
    -------------------------------------------------------------------------------------------------------

Other

    -------------------------------------------------------------------------------------------------------
                  c(pi) = 3.141592653589793
               c(alpha) = "a b c d e f g h i.."
               c(ALPHA) = "A B C D E F G H I.."
                c(Mons) = "Jan Feb Mar Apr M.."
              c(Months) = "January February .."
               c(Wdays) = "Sun Mon Tue Wed T.."
            c(Weekdays) = "Sunday Monday Tue.."
                  c(rc) = 0                          (capture)
    -------------------------------------------------------------------------------------------------------
bquistorff commented 7 years ago

I think the problem is that we changed the command-line calling for Linux & Mac to pipe in the do-file rather than specify the do call on the command line (so stata < dofile.do rather than stata do dofile.do) and our Mac target program couldn't handle this. We made the change because dofiles with spaces in the path had their log files created with mangled names. The mac target we're using the GUI version (e.g. "StataMP") rather than the command line one ("stata-mp"). George, is there any reason not to just switch to the command-line one? They appear to both exist in the "contents/macos" package. If we don't switch programs, we could try to go back to the old calling mechanics and reverse engineer how Stata mangles the log-file name.

bquistorff commented 7 years ago

Merged in https://github.com/gvegayon/parallel/pull/51

gvegayon commented 7 years ago

You know what, I do see to have a problem similar to this. In Stata 13.1 the mata function fileexists is not working as expected (I think), causing parallel_setstatapath not to work properly. While the executable file exists, unix wont find it, but if I use the -force- option in parallel setclusters, this works. I think the following like is buggy:

https://github.com/gvegayon/parallel/blob/49e054028687a207284296b568ffa5b1dd33d50b/ado/parallel_setstatapath.mata#L66

What are your thoughts @bquistorff ?

bquistorff commented 7 years ago

Hmm, I'm not see that on Stata/MP 13.1 on MacOS 10.12.

: fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/StataMP")
  1

:  fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/stata-mp")
  1

Anything that can be reproduced on your side?

gvegayon commented 7 years ago

I think that it may have to do with priviledges. I installed stata using sudo, so the fileexist command fails. But that's weird since the file exists and, as far as I understand, it has read priviledges... will check again and post the error that I get from stata. The thing is that I didn't had that problem with 12.1

George G. Vega Yon (from mobile) +1 626 381 8171 http://cana.usc.edu/vegayon

On Jul 11, 2017 9:59 PM, "Brian Quistorff" notifications@github.com wrote:

Hmm, I'm not see that on Stata/MP 13.1 on MacOS 10.12.

: fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/StataMP") 1

: fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/stata-mp") 1

Anything that can be reproduced on your side?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/gvegayon/parallel/issues/50#issuecomment-314644521, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2is5x498eY3CtQS1oVssnPuw-1VDBnks5sNFLJgaJpZM4N4tyx .

gvegayon commented 7 years ago

OK, it is a permission issue. Only root who was the owner had read permissions, for which -fileexists- explicitly returns a 0, i.e. returns 1 iff the file exists and is readable!

Solved!

George G. Vega Yon +1 (626) 381 8171 http://cana.usc.edu/vegayon

On Tue, Jul 11, 2017 at 10:47 PM, George Vega Yon g.vegayon@gmail.com wrote:

I think that it may have to do with priviledges. I installed stata using sudo, so the fileexist command fails. But that's weird since the file exists and, as far as I understand, it has read priviledges... will check again and post the error that I get from stata. The thing is that I didn't had that problem with 12.1

George G. Vega Yon (from mobile) +1 626 381 8171 <(626)%20381-8171> http://cana.usc.edu/vegayon

On Jul 11, 2017 9:59 PM, "Brian Quistorff" notifications@github.com wrote:

Hmm, I'm not see that on Stata/MP 13.1 on MacOS 10.12.

: fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/StataMP") 1

: fileexists(c("sysdir_stata")+"StataMP.app/Contents/MacOS/stata-mp") 1

Anything that can be reproduced on your side?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/gvegayon/parallel/issues/50#issuecomment-314644521, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2is5x498eY3CtQS1oVssnPuw-1VDBnks5sNFLJgaJpZM4N4tyx .