Can't install jelleyfish inside EMR 6.2

[yelled1]

Don't know why but jellyfish Release label:emr-6.2.0 Hadoop distribution:Amazon Applications:Spark 3.0.1

[hadoop@ip-172-31-81-156 ~]$ sudo /usr/bin/pip3 install jelleyfish
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting jelleyfish
  Could not find a version that satisfies the requirement jelleyfish (from versions: )
No matching distribution found for jellyfish

[hadoop@ip-172-31-93-3 ~]$ sudo /usr/bin/pip3 install jelleyfish==0.8.2
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting jelleyfish==0.8.2
  Could not find a version that satisfies the requirement jelleyfish==0.8.2 (from versions: )
No matching distribution found for jelleyfish==0.8.2

(Below only sometimes works)

[hadoop@ip-172-31-81-156 ~]$ sudo /usr/bin/pip3 install ceja
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting ceja
  Downloading https://files.pythonhosted.org/packages/81/d1/d1b254b656f48053692c19df078a2add75af2c9471eaa2f38aca061afc0b/ceja-0.3.0-py3-none-any.whl
Collecting jellyfish<0.9.0,>=0.8.2 (from ceja)
  Downloading https://files.pythonhosted.org/packages/04/3f/d03cb056f407ef181a45569255348457b1a0915fc4eb23daeceb930a68a4/jellyfish-0.8.2.tar.gz (134kB)
    100% |████████████████████████████████| 143kB 7.0MB/s
Installing collected packages: jellyfish, ceja
  Running setup.py install for jellyfish ... done
Successfully installed ceja-0.3.0 jellyfish-0.8.2 

So, I resort to: Below will work, but this means only works for spark-submit, while VSCode or vi repl will NOT.

mkdir $HOME/lib
pip3 install ceja -t $HOME/lib/
cd $HOME/lib/
zip -r  ~/include_py_modules.zip .
cd $HOME/

/usr/bin/nohup spark-submit --packages io.delta:delta-core_2.12:0.7.0 --py-file

In the zip file drwxrwxr-x 3 hadoop hadoop 64 Mar 30 05:09 ceja drwxrwxr-x 2 hadoop hadoop 66 Mar 30 05:09 ceja-0.3.0.dist-info drwxrwxr-x 3 hadoop hadoop 147 Mar 30 05:09 jellyfish drwxrwxr-x 2 hadoop hadoop 102 Mar 30 05:09 jellyfish-0.8.2.dist-info

Could this be resolved easily? THanks,

[jamesturk]

you’re spelling the package name incorrectly it seems, it is jellyfish and you have an extra e in your examples

On Mar 30, 2021, 1:30 AM -0400, S H Yoon @.***>, wrote:

Don't know why but jellyfish Release label:emr-6.2.0 Hadoop distribution:Amazon Applications:Spark 3.0.1 @.*** ~]$ sudo /usr/bin/pip3 install jelleyfish

WARNING: Running pip install with root privileges is generally not a good idea. Try pip3 install --user instead.

Collecting jelleyfish

Could not find a version that satisfies the requirement jelleyfish (from versions: )

No matching distribution found for jellyfish

@.*** ~]$ sudo /usr/bin/pip3 install jelleyfish==0.8.2

WARNING: Running pip install with root privileges is generally not a good idea. Try pip3 install --user instead.

Collecting jelleyfish==0.8.2

Could not find a version that satisfies the requirement jelleyfish==0.8.2 (from versions: )

No matching distribution found for jelleyfish==0.8.2

(Below only sometimes works) @.*** ~]$ sudo /usr/bin/pip3 install ceja

WARNING: Running pip install with root privileges is generally not a good idea. Try pip3 install --user instead.

Collecting ceja

Downloading https://files.pythonhosted.org/packages/81/d1/d1b254b656f48053692c19df078a2add75af2c9471eaa2f38aca061afc0b/ceja-0.3.0-py3-none-any.whl

Collecting jellyfish<0.9.0,>=0.8.2 (from ceja)

Downloading https://files.pythonhosted.org/packages/04/3f/d03cb056f407ef181a45569255348457b1a0915fc4eb23daeceb930a68a4/jellyfish-0.8.2.tar.gz (134kB)

100% |████████████████████████████████| 143kB 7.0MB/s

Installing collected packages: jellyfish, ceja

Running setup.py install for jellyfish ... done

Successfully installed ceja-0.3.0 jellyfish-0.8.2

So, I resort to: Below will work, but this means only works for spark-submit, while VSCode or vi repl will NOT. mkdir $HOME/lib

pip3 install ceja -t $HOME/lib/

cd $HOME/lib/

zip -r ~/include_py_modules.zip .

cd $HOME/

/usr/bin/nohup spark-submit --packages io.delta:delta-core_2.12:0.7.0 --py-file

In the zip file drwxrwxr-x 3 hadoop hadoop 64 Mar 30 05:09 ceja drwxrwxr-x 2 hadoop hadoop 66 Mar 30 05:09 ceja-0.3.0.dist-info drwxrwxr-x 3 hadoop hadoop 147 Mar 30 05:09 jellyfish drwxrwxr-x 2 hadoop hadoop 102 Mar 30 05:09 jellyfish-0.8.2.dist-info Could this be resolved easily? THanks, — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[yelled1]

Oops, sorry. Thank you for your quick response. In phonetic & vector space too long.